2024-03-25T18:03:21.101 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:03:21.101 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/orig.config.yaml 2024-03-25T18:03:21.142 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:03:21.231 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621320' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFN2YZrooIvVsISkt9MTFOIWr/dG0HSHOvYeYXmO+8d07avH0ttJ1lX1LOopxikblA5JwqPr0bIC6G6CEjmTZ5I= smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFignQ13rXWoi83NQO6kOItIIJTX6k/evnxiJbbCD4HIK+LWHM0J7ITDwhp08F5/ptql0cVFiEQLX4gfn4ga9j4= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:03:21.231 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:03:21.232 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:03:21.232 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:03:21.233 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:03:21.255 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:03:21.255 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:03:21.256 INFO:teuthology.packaging:ref: None 2024-03-25T18:03:21.256 INFO:teuthology.packaging:tag: None 2024-03-25T18:03:21.256 INFO:teuthology.packaging:branch: main 2024-03-25T18:03:21.256 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:03:21.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:03:21.475 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:03:21.477 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:03:21.484 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:03:21.484 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:03:21.491 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:03:21.505 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:03:21.511 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:03:21.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:58:27.344015', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFN2YZrooIvVsISkt9MTFOIWr/dG0HSHOvYeYXmO+8d07avH0ttJ1lX1LOopxikblA5JwqPr0bIC6G6CEjmTZ5I='} 2024-03-25T18:03:21.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:58:27.345894', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFignQ13rXWoi83NQO6kOItIIJTX6k/evnxiJbbCD4HIK+LWHM0J7ITDwhp08F5/ptql0cVFiEQLX4gfn4ga9j4='} 2024-03-25T18:03:21.551 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:03:21.557 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:03:21.557 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:03:21.557 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:03:21.627 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3be47970d0>, signals=[15]) 2024-03-25T18:03:21.628 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:03:21.633 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:03:21.634 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T18:03:21.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:03:21.742 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T18:03:21.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:03:21.830 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:03:21.837 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-03-25T18:03:21.854 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-03-25T18:03:21.854 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-03-25T18:03:21.910 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-03-25T18:03:21.910 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="9" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="9" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:03:21.911 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:03:21.912 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-03-25T18:03:21.930 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-03-25T18:03:21.947 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-03-25T18:03:21.947 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:03:22.004 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:03:22.005 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-03-25T18:03:22.022 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:03:22.030 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:03:22.035 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:03:22.036 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:03:22.038 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:03:22.059 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:03:22.064 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:03:22.064 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:03:22.095 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:03:22.216 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:03:22.274 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:03:22.289 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:03:22.289 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:03:22.290 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:03:22.290 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:03:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:03:22.307 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-03-25T18:03:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:03:22.308 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-03-25T18:03:22.364 INFO:teuthology.orchestra.run.smithi142.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:03:22.364 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.14.0-361.el9.x86_64 2024-03-25T18:03:22.364 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum install -y kernel 2024-03-25T18:03:22.365 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:03:22.366 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.14.0-361.el9.x86_64 2024-03-25T18:03:22.366 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2024-03-25T18:03:24.426 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:03:24.552 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T18:03:25.910 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T18:03:25.952 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T18:03:33.166 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 1.2 MB/s | 8.0 MB 00:06 2024-03-25T18:03:34.592 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 976 kB/s | 8.0 MB 00:08 2024-03-25T18:03:41.777 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-03-25T18:03:42.368 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T18:03:53.365 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 774 kB/s | 6.1 MB 00:08 2024-03-25T18:03:53.931 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 789 kB/s | 6.1 MB 00:07 2024-03-25T18:03:55.086 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-25T18:03:56.236 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T18:03:56.787 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T18:03:58.554 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 21 MB 00:02 2024-03-25T18:04:01.704 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T18:04:03.745 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-25T18:04:03.835 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:04:03.836 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:04:03.925 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:04:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout:Install 4 Packages 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:04:03.930 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 76 M 2024-03-25T18:04:03.931 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 123 M 2024-03-25T18:04:03.931 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:04:04.111 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:04.111 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:04.111 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:04.250 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:04.259 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:04.282 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:04.411 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:04.418 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:04.441 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:04.764 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:04.782 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:04.804 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:04.809 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:04.823 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:04.879 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:04.899 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:04.899 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:04.924 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:05.369 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:05.391 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:05.425 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:05.688 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:05.723 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:05.741 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:05.807 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:05.824 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:05.850 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:05.868 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:05.889 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:05.906 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:05.925 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:05.935 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:04:05.936 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:04:05.943 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:06.011 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:06.037 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:04:06.044 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:Install 4 Packages 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 76 M 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 123 M 2024-03-25T18:04:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:04:06.072 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:06.086 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:06.090 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:06.124 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:06.164 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:06.164 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:06.181 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:06.182 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:06.216 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:06.233 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:06.248 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:06.265 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:06.285 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:06.299 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:06.355 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:06.355 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:06.356 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:06.366 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:06.425 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:06.460 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:06.471 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:06.472 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:06.491 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:06.496 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:06.587 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:06.622 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:06.657 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:06.965 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:06.967 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:07.002 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:07.005 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:07.008 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:07.032 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:07.050 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:07.051 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:07.094 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:07.115 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:07.116 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:07.139 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:07.263 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.291 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.299 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:07.301 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.339 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:07.352 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.363 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:07.364 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.387 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.389 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.421 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.427 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.453 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:07.456 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:07.464 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:07.473 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:07.473 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:07.481 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:07.508 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:07.509 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:07.515 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:07.534 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.543 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.568 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.615 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:07.616 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:07.676 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:07.685 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.712 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.726 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.738 INFO:teuthology.orchestra.run.smithi142.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 18 kB/s | 66 kB 00:03 2024-03-25T18:04:07.775 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.815 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.827 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:07.892 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:07.933 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:07.940 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:07.953 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:08.049 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:08.138 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:08.175 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:08.202 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:08.329 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:08.351 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:08.384 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:08.463 INFO:teuthology.orchestra.run.smithi142.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.4 MB/s | 15 MB 00:04 2024-03-25T18:04:08.553 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:08.697 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:08.719 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:08.743 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:08.852 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:08.852 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:08.872 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:08.872 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:08.891 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:08.897 INFO:teuthology.orchestra.run.smithi142.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.9 MB/s | 33 MB 00:04 2024-03-25T18:04:08.910 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:08.913 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:08.917 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:08.976 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:08.987 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:09.009 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:09.009 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:09.026 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.026 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.035 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:09.080 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:09.098 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.103 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.107 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.128 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:09.156 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:09.156 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:09.157 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.174 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:09.317 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:09.326 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:09.334 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:09.455 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:09.668 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:09.693 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:09.701 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:09.720 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:09.751 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:09.752 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:09.877 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:09.912 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:09.920 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:09.972 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.981 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.981 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:09.989 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.989 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:09.997 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:10.006 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:10.063 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:10.070 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.077 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.083 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.124 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:10.225 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.236 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.267 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.287 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.345 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:10.346 INFO:teuthology.orchestra.run.smithi047.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 16 kB/s | 66 kB 00:04 2024-03-25T18:04:10.400 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:10.418 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:10.435 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:10.476 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:10.536 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:10.569 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:10.707 INFO:teuthology.orchestra.run.smithi047.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.4 MB/s | 15 MB 00:04 2024-03-25T18:04:10.730 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:10.864 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:10.905 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:10.969 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:11.299 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:11.448 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:11.524 INFO:teuthology.orchestra.run.smithi047.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.4 MB/s | 33 MB 00:05 2024-03-25T18:04:11.536 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:04:11.599 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:11.675 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:11.710 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:11.779 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:04:11.946 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:12.028 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:04:12.117 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:12.260 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:04:12.380 INFO:teuthology.orchestra.run.smithi142.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.9 MB/s | 27 MB 00:04 2024-03-25T18:04:12.386 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:12.386 INFO:teuthology.orchestra.run.smithi142.stdout:Total 9.0 MB/s | 76 MB 00:08 2024-03-25T18:04:12.393 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:12.459 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:04:12.811 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:12.973 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:04:13.049 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:04:13.057 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:13.113 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:04:13.214 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:13.259 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:04:13.317 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:13.335 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:13.471 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:04:13.471 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:04:13.507 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:13.543 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:04:13.576 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:13.932 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:04:14.000 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:04:14.080 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:04:14.154 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:04:14.489 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:04:14.489 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:04:14.824 INFO:teuthology.orchestra.run.smithi047.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.1 MB/s | 27 MB 00:04 2024-03-25T18:04:14.829 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:14.829 INFO:teuthology.orchestra.run.smithi047.stdout:Total 8.6 MB/s | 76 MB 00:08 2024-03-25T18:04:15.520 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:04:15.971 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:04:15.971 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:04:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:04:17.043 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:04:17.044 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:04:17.576 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:04:17.675 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:04:19.656 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:04:20.281 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:04:20.374 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:04:20.476 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:04:20.620 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:04:23.280 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:04:23.417 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:04:25.324 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:04:25.404 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:28.483 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:04:28.525 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:30.121 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:33.632 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:59.253 INFO:teuthology.orchestra.run.smithi142.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:04:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:04:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:59.728 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:04:59.728 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:04:59.728 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:04:59.728 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:05:00.145 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:05:00.145 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:05:00.145 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:05:00.146 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:05:00.146 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:05:00.146 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:05:00.146 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:05:00.146 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:05:00.146 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:05:00.472 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:05:00.589 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:05:00.589 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:05:00.589 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:05:00.589 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:05:00.589 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:05:00.589 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:05:00.590 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:05:00.590 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:05:00.670 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:05:00.670 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:05:00.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo rpm -qi grub2-tools 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Name : grub2-tools 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Epoch : 1 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Version : 2.06 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Release : 68.el9 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: x86_64 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Group : Unspecified 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Size : 8264254 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:License : GPLv3+ 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:05:00.712 INFO:teuthology.orchestra.run.smithi142.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:Packager : builder@centos.org 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:Vendor : CentOS 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:Summary : Support tools for GRUB. 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:Description : 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:hardware devices. 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:05:00.713 INFO:teuthology.orchestra.run.smithi142.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:05:00.714 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:05:00.714 INFO:teuthology.task.kernel:Updating grub on smithi142 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:05:00.714 DEBUG:teuthology.orchestra.run.smithi142:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:05:01.349 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2024-03-25T18:05:02.746 INFO:teuthology.orchestra.run.smithi142.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:05:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:done 2024-03-25T18:05:02.769 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-03-25T18:05:02.784 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.SqsYpCbtG3 2024-03-25T18:05:02.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SqsYpCbtG3 2024-03-25T18:05:02.854 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.SqsYpCbtG3 2024-03-25T18:05:02.990 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.SqsYpCbtG3 is 6KB 2024-03-25T18:05:03.039 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.SqsYpCbtG3 2024-03-25T18:05:03.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:05:03.128 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:05:03.128 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:05:03.128 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:05:03.128 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:05:03.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:05:03.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2024-03-25T18:05:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:05:04.403 INFO:teuthology.orchestra.run.smithi047.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:05:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:05:04.450 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:05:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:05:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:05:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:05:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:05:05.237 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:05:05.238 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:05:05.238 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:05:05.539 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:05:05.648 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:05:05.648 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:05:05.648 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:05:05.649 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:05:05.649 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:05:05.649 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:05:05.649 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:05:05.649 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:05:05.731 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:05:05.732 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:05:05.732 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.06 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 68.el9 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Group : Unspecified 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 8264254 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:05:05.770 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : builder@centos.org 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : CentOS 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:05:05.771 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:05:05.772 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:05:05.772 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:05:05.772 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:05:06.490 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-03-25T18:05:07.933 INFO:teuthology.orchestra.run.smithi047.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:05:07.953 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-03-25T18:05:07.956 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-25T18:05:08.011 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.uUBdATCcv6 2024-03-25T18:05:08.011 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uUBdATCcv6 2024-03-25T18:05:08.083 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.uUBdATCcv6 2024-03-25T18:05:08.220 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.uUBdATCcv6 is 6KB 2024-03-25T18:05:08.241 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.uUBdATCcv6 2024-03-25T18:05:08.256 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:05:08.324 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:05:08.324 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:05:08.324 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:05:08.324 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:05:08.326 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:05:08.481 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-03-25T18:05:33.314 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:05:33.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-25T18:05:33.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:05:38.486 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:05:38.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-25T18:05:38.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:05:51.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-25T18:05:56.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-03-25T18:06:00.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-25T18:06:00.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:06:05.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-25T18:06:05.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:07:00.834 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:07:05.908 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:07:12.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-25T18:07:12.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:07:13.326 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-03-25T18:07:13.695 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-25T18:07:13.695 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:07:13.695 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-03-25T18:07:13.711 INFO:teuthology.orchestra.run.smithi142.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:07:13.711 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:07:13.711 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:07:13.711 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:07:14.712 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:07:14.713 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:07:14.776 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2024-03-25T18:07:14.813 DEBUG:teuthology.parallel:result is None 2024-03-25T18:07:17.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-25T18:07:17.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:07:18.243 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-03-25T18:07:18.648 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-25T18:07:18.648 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:07:18.648 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-03-25T18:07:18.663 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:07:18.663 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:07:18.663 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:07:18.663 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:07:19.664 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:07:19.664 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:07:19.721 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-03-25T18:07:19.764 DEBUG:teuthology.parallel:result is None 2024-03-25T18:07:19.764 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:07:19.778 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:07:19.779 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:07:19.781 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:07:19.798 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:07:19.804 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:07:19.816 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:07:19.817 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:07:19.838 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:07:19.904 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:07:19.910 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:07:19.911 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:07:19.913 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:07:19.946 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:07:19.947 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:07:19.960 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:07:19.961 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:07:19.963 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:07:19.970 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:07:19.970 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:07:20.004 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:07:20.034 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:07:20.043 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:07:20.044 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:07:20.075 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:07:20.091 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:07:20.170 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:07:20.216 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:07:20.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:07:20.341 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T18:07:20.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:07:20.409 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:07:20.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:07:20.508 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:07:20.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:07:20.637 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T18:07:20.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:07:20.707 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-03-25T18:07:20.709 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-03-25T18:07:20.747 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:07:20.801 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:07:21.235 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:07:21.244 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:07:21.244 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:07:21.267 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:07:21.315 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-03-25T18:07:21.370 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-03-25T18:07:21.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2024-03-25T18:07:21.439 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: 2024-03-25T18:07:21.441 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:07:21.441 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-03-25T18:07:21.459 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-03-25T18:07:21.460 DEBUG:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2024-03-25T18:07:21.480 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2024-03-25T18:07:21.480 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi142.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:07:21.480 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:07:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:07:21.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:07:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:07:21.558 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:07:21.558 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:07:21.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:07:21.640 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:07:21.650 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T18:07:21.655 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T18:07:21.796 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:07:21.811 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:07:21.812 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:12:09.992 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2024-03-25T18:12:09.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-25T18:12:09.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:12:10.083 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-03-25T18:12:10.173 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-25T18:12:10.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-25T18:12:10.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:12:10.261 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-03-25T18:12:10.350 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-25T18:12:10.350 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:12:10.362 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:12:10.362 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:12:10.362 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:12:10.364 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:12:10.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:12:10.393 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:12:10.410 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:12:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:12:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:12:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-03-25T18:12:10.454 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-03-25T18:12:10.467 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2024-03-25T18:12:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:12:10.480 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2024-03-25T18:12:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:12:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:12:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:12:10.544 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:12:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:12:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-03-25T18:12:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.570 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:12:10.574 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:12:10.574 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-03-25T18:12:10.574 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.574 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.574 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.574 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:12:10.574 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:12:10.583 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:12:10.583 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:12:10.583 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:12:10.583 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:12:10.588 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:12:10.588 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:12:10.589 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:12:10.589 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:12:10.591 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:12:10.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:12:10.736 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:12:10.737 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:12:10.743 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:12:10.744 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:12:11.086 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:12:11.086 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T18:12:11.087 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:12:11.114 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:12:11.114 DEBUG:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:12:11.139 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:12:11.139 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T18:12:11.139 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:12:11.168 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:12:11.168 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:12:11.182 DEBUG:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:12:11.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:12:11.261 DEBUG:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:12:11.317 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:12:11.328 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2024-03-25T18:12:11.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum clean all 2024-03-25T18:12:11.385 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2024-03-25T18:12:11.386 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2024-03-25T18:12:11.570 INFO:teuthology.orchestra.run.smithi142.stdout:74 files removed 2024-03-25T18:12:11.597 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw 2024-03-25T18:12:11.646 INFO:teuthology.orchestra.run.smithi047.stdout:74 files removed 2024-03-25T18:12:11.672 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw 2024-03-25T18:12:13.038 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 72 kB/s | 82 kB 00:01 2024-03-25T18:12:13.134 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 71 kB/s | 82 kB 00:01 2024-03-25T18:12:14.026 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:12:14.638 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 6.5 kB/s | 10 kB 00:01 2024-03-25T18:12:14.793 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-25T18:12:15.295 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:12:15.422 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-25T18:12:15.857 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T18:12:16.862 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-25T18:12:17.375 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-25T18:12:28.061 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 796 kB/s | 8.0 MB 00:10 2024-03-25T18:12:28.700 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 717 kB/s | 8.0 MB 00:11 2024-03-25T18:12:51.839 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 877 kB/s | 19 MB 00:22 2024-03-25T18:12:52.885 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 814 kB/s | 19 MB 00:23 2024-03-25T18:13:06.509 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 570 kB/s | 6.1 MB 00:11 2024-03-25T18:13:07.913 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 543 kB/s | 6.1 MB 00:11 2024-03-25T18:13:07.930 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-03-25T18:13:08.956 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T18:13:09.702 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-25T18:13:10.769 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T18:13:13.941 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-25T18:13:15.635 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-25T18:13:16.457 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options 2024-03-25T18:13:16.465 INFO:teuthology.orchestra.run.smithi047.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:13:16.466 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout:Installing weak dependencies: 2024-03-25T18:13:16.467 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:13:16.468 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:13:16.468 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:13:16.468 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:13:16.468 INFO:teuthology.orchestra.run.smithi047.stdout:Install 30 Packages 2024-03-25T18:13:16.468 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 2 Packages 2024-03-25T18:13:16.468 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:13:16.469 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 58 M 2024-03-25T18:13:16.469 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:13:18.077 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options 2024-03-25T18:13:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:13:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:13:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:Installing weak dependencies: 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:Install 30 Packages 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 2 Packages 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 58 M 2024-03-25T18:13:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:13:34.582 INFO:teuthology.orchestra.run.smithi047.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 308 kB/s | 5.3 MB 00:17 2024-03-25T18:13:34.840 INFO:teuthology.orchestra.run.smithi047.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 97 kB/s | 25 kB 00:00 2024-03-25T18:13:36.634 INFO:teuthology.orchestra.run.smithi142.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 299 kB/s | 5.3 MB 00:18 2024-03-25T18:13:36.759 INFO:teuthology.orchestra.run.smithi142.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 201 kB/s | 25 kB 00:00 2024-03-25T18:13:38.057 INFO:teuthology.orchestra.run.smithi047.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 231 kB/s | 742 kB 00:03 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi142.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 270 kB/s | 742 kB 00:02 2024-03-25T18:13:40.056 INFO:teuthology.orchestra.run.smithi047.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 249 kB/s | 498 kB 00:01 2024-03-25T18:13:41.326 INFO:teuthology.orchestra.run.smithi142.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 274 kB/s | 498 kB 00:01 2024-03-25T18:13:49.767 INFO:teuthology.orchestra.run.smithi142.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 317 kB/s | 9.7 MB 00:31 2024-03-25T18:13:50.373 INFO:teuthology.orchestra.run.smithi047.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 297 kB/s | 9.7 MB 00:33 2024-03-25T18:13:50.483 INFO:teuthology.orchestra.run.smithi142.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 63 kB/s | 45 kB 00:00 2024-03-25T18:13:50.590 INFO:teuthology.orchestra.run.smithi047.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 208 kB/s | 45 kB 00:00 2024-03-25T18:13:51.273 INFO:teuthology.orchestra.run.smithi047.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 192 kB/s | 131 kB 00:00 2024-03-25T18:13:51.450 INFO:teuthology.orchestra.run.smithi142.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 136 kB/s | 131 kB 00:00 2024-03-25T18:13:52.140 INFO:teuthology.orchestra.run.smithi047.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 187 kB/s | 162 kB 00:00 2024-03-25T18:13:52.325 INFO:teuthology.orchestra.run.smithi142.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 185 kB/s | 162 kB 00:00 2024-03-25T18:13:53.209 INFO:teuthology.orchestra.run.smithi142.stdout:(9/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 584 kB/s | 20 MB 00:34 2024-03-25T18:13:53.882 INFO:teuthology.orchestra.run.smithi047.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 184 kB/s | 320 kB 00:01 2024-03-25T18:13:54.175 INFO:teuthology.orchestra.run.smithi142.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 173 kB/s | 320 kB 00:01 2024-03-25T18:13:54.542 INFO:teuthology.orchestra.run.smithi142.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 223 kB/s | 298 kB 00:01 2024-03-25T18:13:54.593 INFO:teuthology.orchestra.run.smithi142.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 238 kB/s | 99 kB 00:00 2024-03-25T18:13:54.726 INFO:teuthology.orchestra.run.smithi142.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 577 kB/s | 106 kB 00:00 2024-03-25T18:13:54.751 INFO:teuthology.orchestra.run.smithi142.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T18:13:54.776 INFO:teuthology.orchestra.run.smithi142.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-25T18:13:54.844 INFO:teuthology.orchestra.run.smithi142.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-25T18:13:54.869 INFO:teuthology.orchestra.run.smithi142.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-25T18:13:54.894 INFO:teuthology.orchestra.run.smithi142.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-25T18:13:55.028 INFO:teuthology.orchestra.run.smithi142.stdout:(19/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 381 kB/s | 5.1 MB 00:13 2024-03-25T18:13:55.108 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:13:55.109 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:13:55.187 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:13:55.231 INFO:teuthology.orchestra.run.smithi047.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 221 kB/s | 298 kB 00:01 2024-03-25T18:13:55.458 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:13:55.486 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:13:55.534 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:13:55.540 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:13:55.561 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:13:55.599 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:13:55.632 INFO:teuthology.orchestra.run.smithi047.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 248 kB/s | 99 kB 00:00 2024-03-25T18:13:55.688 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:13:55.700 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:13:55.722 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:13:55.734 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:13:55.740 INFO:teuthology.orchestra.run.smithi047.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T18:13:55.762 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:13:55.779 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:13:55.791 INFO:teuthology.orchestra.run.smithi047.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:13:55.807 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:55.807 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:55.816 INFO:teuthology.orchestra.run.smithi047.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.8 MB/s | 44 kB 00:00 2024-03-25T18:13:55.821 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:55.875 INFO:teuthology.orchestra.run.smithi047.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:13:55.916 INFO:teuthology.orchestra.run.smithi047.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:13:55.925 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:55.925 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:55.942 INFO:teuthology.orchestra.run.smithi047.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T18:13:55.943 INFO:teuthology.orchestra.run.smithi142.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:55.978 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:13:56.004 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T18:13:56.096 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T18:13:56.196 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:13:56.329 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:13:56.356 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:13:56.411 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:13:56.452 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:56.536 INFO:teuthology.orchestra.run.smithi142.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 22 kB/s | 36 kB 00:01 2024-03-25T18:13:56.541 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:56.570 INFO:teuthology.orchestra.run.smithi142.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 31 kB/s | 47 kB 00:01 2024-03-25T18:13:56.588 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:13:56.686 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:13:56.862 INFO:teuthology.orchestra.run.smithi142.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 111 kB/s | 221 kB 00:01 2024-03-25T18:13:57.079 INFO:teuthology.orchestra.run.smithi142.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-25T18:13:57.100 INFO:teuthology.orchestra.run.smithi047.stdout:(18/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 306 kB/s | 5.1 MB 00:17 2024-03-25T18:13:57.118 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:13:57.142 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T18:13:57.179 INFO:teuthology.orchestra.run.smithi142.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-25T18:13:57.204 INFO:teuthology.orchestra.run.smithi142.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 76 kB/s | 26 kB 00:00 2024-03-25T18:13:57.248 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T18:13:57.283 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:13:57.287 INFO:teuthology.orchestra.run.smithi142.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-25T18:13:57.315 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:13:57.333 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:13:57.389 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:13:57.409 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:57.429 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:57.438 INFO:teuthology.orchestra.run.smithi142.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-03-25T18:13:57.452 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:13:57.463 INFO:teuthology.orchestra.run.smithi142.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-25T18:13:57.475 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:13:57.563 INFO:teuthology.orchestra.run.smithi142.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-25T18:13:57.626 INFO:teuthology.orchestra.run.smithi047.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 131 kB/s | 221 kB 00:01 2024-03-25T18:13:57.639 INFO:teuthology.orchestra.run.smithi142.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-03-25T18:13:57.643 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:13:57.663 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T18:13:57.737 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T18:13:57.843 INFO:teuthology.orchestra.run.smithi047.stdout:(20/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 496 kB/s | 20 MB 00:41 2024-03-25T18:13:57.843 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:13:57.878 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:13:57.895 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:13:57.953 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:13:57.974 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:57.994 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:13:58.017 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:13:58.061 INFO:teuthology.orchestra.run.smithi047.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:13:58.443 INFO:teuthology.orchestra.run.smithi047.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 515 kB/s | 309 kB 00:00 2024-03-25T18:13:58.460 INFO:teuthology.orchestra.run.smithi047.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 26 kB/s | 36 kB 00:01 2024-03-25T18:13:58.693 INFO:teuthology.orchestra.run.smithi047.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 44 kB/s | 47 kB 00:01 2024-03-25T18:13:58.710 INFO:teuthology.orchestra.run.smithi047.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-03-25T18:13:58.902 INFO:teuthology.orchestra.run.smithi047.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-03-25T18:13:58.936 INFO:teuthology.orchestra.run.smithi047.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 299 kB/s | 67 kB 00:00 2024-03-25T18:13:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-25T18:13:59.045 INFO:teuthology.orchestra.run.smithi047.stdout:(28/32): liboath-2.6.7-2.el9.x86_64.rpm 138 kB/s | 49 kB 00:00 2024-03-25T18:13:59.162 INFO:teuthology.orchestra.run.smithi047.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-25T18:13:59.187 INFO:teuthology.orchestra.run.smithi047.stdout:(30/32): re2-20211101-3.el9.x86_64.rpm 765 kB/s | 192 kB 00:00 2024-03-25T18:14:04.721 INFO:teuthology.orchestra.run.smithi142.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 449 kB/s | 3.1 MB 00:07 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi047.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 421 kB/s | 3.4 MB 00:08 2024-03-25T18:14:08.586 INFO:teuthology.orchestra.run.smithi047.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 340 kB/s | 3.1 MB 00:09 2024-03-25T18:14:08.591 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:14:08.592 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.1 MB/s | 58 MB 00:52 2024-03-25T18:14:08.625 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:14:08.667 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:14:08.668 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:14:09.156 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:14:09.157 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:14:09.204 INFO:teuthology.orchestra.run.smithi142.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 294 kB/s | 3.4 MB 00:11 2024-03-25T18:14:09.214 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:14:09.214 INFO:teuthology.orchestra.run.smithi142.stdout:Total 1.1 MB/s | 58 MB 00:51 2024-03-25T18:14:09.246 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:14:09.284 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:14:09.284 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:14:09.758 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:14:09.759 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:14:10.660 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:14:10.745 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:14:10.782 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:14:10.808 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:14:10.830 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:14:10.847 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:14:11.040 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:14:11.096 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:14:11.232 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:14:11.316 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:14:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:14:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:14:11.415 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:14:11.438 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:14:11.509 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:14:11.575 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:14:11.602 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:14:11.630 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:14:11.683 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:14:11.711 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:14:11.756 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:14:11.786 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:14:11.830 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:14:11.899 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:14:11.922 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:14:11.949 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:14:11.994 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:14:12.021 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:14:12.028 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:14:12.111 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:14:12.139 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:14:12.171 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:14:12.199 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:14:12.212 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:14:12.231 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:14:12.276 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:14:12.305 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:14:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:14:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:14:12.425 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:14:12.449 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:14:12.449 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:14:12.481 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:14:12.513 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:14:12.548 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:14:12.654 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:14:12.699 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:14:12.739 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:14:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:14:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:14:12.826 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:14:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:14:12.861 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:14:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:14:12.966 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:14:13.005 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:14:13.146 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:14:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:14:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:14:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:14:13.339 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:14:13.485 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:14:13.522 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:14:13.576 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:14:13.647 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:14:13.682 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:14:13.799 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:14:13.825 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:14:13.904 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:14:13.971 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:14:14.002 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:14:15.931 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:14:16.195 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:14:16.240 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:14:16.598 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:14:16.668 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:14:16.668 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:14:16.668 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:14:16.668 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:16.725 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:14:16.803 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:14:16.843 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:14:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:14:16.927 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:14:16.927 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:14:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:14:26.274 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /root 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2024-03-25T18:14:26.275 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /root 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2024-03-25T18:14:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:27.204 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:14:27.228 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:14:27.228 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:14:27.228 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:14:27.228 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:14:27.228 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:14:27.228 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:27.230 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:14:27.316 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:14:27.316 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:14:27.544 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:14:27.568 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:14:27.568 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:14:27.568 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:14:27.568 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:14:27.569 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:14:27.569 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:27.571 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:14:27.655 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:14:27.655 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:14:28.497 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:14:28.498 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:14:28.499 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:14:28.500 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:14:28.736 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:14:28.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:14:28.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:14:28.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:14:28.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:14:28.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:14:28.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:14:28.739 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:14:29.478 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:14:29.479 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.480 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:29.481 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:14:29.817 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-test 2024-03-25T18:14:29.922 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:14:29.922 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:29.922 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2024-03-25T18:14:29.922 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.923 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:29.924 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:14:29.925 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:14:29.925 INFO:teuthology.orchestra.run.smithi047.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:14:29.925 INFO:teuthology.orchestra.run.smithi047.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:14:29.925 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:14:29.925 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:29.925 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:14:30.255 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test 2024-03-25T18:14:30.378 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:14:30.534 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:14:30.535 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:14:30.535 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout:Install 6 Packages 2024-03-25T18:14:30.536 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:14:30.537 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 48 M 2024-03-25T18:14:30.537 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 204 M 2024-03-25T18:14:30.537 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:14:30.811 INFO:teuthology.orchestra.run.smithi142.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T18:14:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:14:30.878 INFO:teuthology.orchestra.run.smithi142.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.7 MB/s | 300 kB 00:00 2024-03-25T18:14:30.903 INFO:teuthology.orchestra.run.smithi142.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T18:14:30.984 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:14:30.985 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-03-25T18:14:30.986 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:14:30.987 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 48 M 2024-03-25T18:14:30.987 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 204 M 2024-03-25T18:14:30.987 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:14:31.201 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:14:31.276 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T18:14:31.309 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:14:31.493 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T18:14:31.504 INFO:teuthology.orchestra.run.smithi142.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 193 kB/s | 163 kB 00:00 2024-03-25T18:14:31.737 INFO:teuthology.orchestra.run.smithi142.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 224 kB/s | 187 kB 00:00 2024-03-25T18:14:31.801 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 210 kB/s | 163 kB 00:00 2024-03-25T18:16:20.940 INFO:teuthology.orchestra.run.smithi142.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 439 kB/s | 47 MB 01:50 2024-03-25T18:16:20.942 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:20.942 INFO:teuthology.orchestra.run.smithi142.stdout:Total 447 kB/s | 48 MB 01:50 2024-03-25T18:16:20.953 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:16:20.973 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:16:20.973 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:16:21.366 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:16:21.366 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:16:21.970 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:16:22.047 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:16:22.096 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:16:22.135 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:16:22.186 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:16:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:16:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:16:28.523 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 412 kB/s | 47 MB 01:57 2024-03-25T18:16:28.525 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:28.526 INFO:teuthology.orchestra.run.smithi047.stdout:Total 420 kB/s | 48 MB 01:57 2024-03-25T18:16:28.537 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:16:28.558 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:16:28.558 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:16:28.954 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:16:28.955 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:16:29.361 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:16:29.362 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:16:29.362 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:16:29.362 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:16:29.362 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:16:29.362 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:16:29.559 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:16:29.633 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:16:29.669 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:16:29.709 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:16:29.749 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:16:29.776 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:16:30.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph 2024-03-25T18:16:30.677 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:03:15 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:16:30.833 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout:========================================================================================= 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout:========================================================================================= 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:16:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:16:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:16:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout:========================================================================================= 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout:Install 36 Packages 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 30 M 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 107 M 2024-03-25T18:16:30.839 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:16:31.809 INFO:teuthology.orchestra.run.smithi142.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:16:34.474 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:16:34.561 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:16:34.792 INFO:teuthology.orchestra.run.smithi142.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 446 kB/s | 1.5 MB 00:03 2024-03-25T18:16:36.896 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:16:36.896 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:16:36.896 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:16:36.896 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:16:36.896 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:16:36.896 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:16:36.950 INFO:teuthology.orchestra.run.smithi142.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 389 kB/s | 2.1 MB 00:05 2024-03-25T18:16:37.308 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:16:37.308 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:16:37.309 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:16:37.635 INFO:teuthology.orchestra.run.smithi142.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 362 kB/s | 247 kB 00:00 2024-03-25T18:16:37.660 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph 2024-03-25T18:16:37.759 INFO:teuthology.orchestra.run.smithi142.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:16:37.801 INFO:teuthology.orchestra.run.smithi142.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:16:37.826 INFO:teuthology.orchestra.run.smithi142.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:16:37.852 INFO:teuthology.orchestra.run.smithi142.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:16:37.894 INFO:teuthology.orchestra.run.smithi142.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-25T18:16:37.936 INFO:teuthology.orchestra.run.smithi142.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:16:38.011 INFO:teuthology.orchestra.run.smithi142.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:16:38.044 INFO:teuthology.orchestra.run.smithi142.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:16:38.078 INFO:teuthology.orchestra.run.smithi142.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:16:38.120 INFO:teuthology.orchestra.run.smithi142.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:16:38.233 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:03:25 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:16:38.397 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:16:38.399 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:16:38.400 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:16:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:16:38.402 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-25T18:16:38.403 INFO:teuthology.orchestra.run.smithi047.stdout:Install 36 Packages 2024-03-25T18:16:38.403 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:16:38.403 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 30 M 2024-03-25T18:16:38.403 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 107 M 2024-03-25T18:16:38.403 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:16:38.879 INFO:teuthology.orchestra.run.smithi142.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-03-25T18:16:38.971 INFO:teuthology.orchestra.run.smithi142.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-25T18:16:39.229 INFO:teuthology.orchestra.run.smithi142.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 585 kB/s | 151 kB 00:00 2024-03-25T18:16:39.288 INFO:teuthology.orchestra.run.smithi142.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 745 kB/s | 43 kB 00:00 2024-03-25T18:16:39.377 INFO:teuthology.orchestra.run.smithi047.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T18:16:39.421 INFO:teuthology.orchestra.run.smithi142.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-25T18:16:39.613 INFO:teuthology.orchestra.run.smithi142.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T18:16:39.664 INFO:teuthology.orchestra.run.smithi142.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-25T18:16:39.714 INFO:teuthology.orchestra.run.smithi142.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-25T18:16:39.764 INFO:teuthology.orchestra.run.smithi142.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-03-25T18:16:39.815 INFO:teuthology.orchestra.run.smithi142.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-25T18:16:39.865 INFO:teuthology.orchestra.run.smithi142.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 396 kB/s | 20 kB 00:00 2024-03-25T18:16:39.915 INFO:teuthology.orchestra.run.smithi142.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-03-25T18:16:39.990 INFO:teuthology.orchestra.run.smithi142.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T18:16:40.124 INFO:teuthology.orchestra.run.smithi142.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-25T18:16:40.183 INFO:teuthology.orchestra.run.smithi142.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-25T18:16:40.241 INFO:teuthology.orchestra.run.smithi142.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T18:16:40.308 INFO:teuthology.orchestra.run.smithi142.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 538 kB/s | 36 kB 00:00 2024-03-25T18:16:40.442 INFO:teuthology.orchestra.run.smithi142.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-25T18:16:40.609 INFO:teuthology.orchestra.run.smithi142.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-25T18:16:40.659 INFO:teuthology.orchestra.run.smithi142.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-25T18:16:41.444 INFO:teuthology.orchestra.run.smithi047.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 603 kB/s | 1.5 MB 00:02 2024-03-25T18:16:42.894 INFO:teuthology.orchestra.run.smithi047.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 546 kB/s | 2.1 MB 00:03 2024-03-25T18:16:43.552 INFO:teuthology.orchestra.run.smithi047.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 376 kB/s | 247 kB 00:00 2024-03-25T18:16:43.677 INFO:teuthology.orchestra.run.smithi047.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:16:43.719 INFO:teuthology.orchestra.run.smithi047.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:16:43.753 INFO:teuthology.orchestra.run.smithi047.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-25T18:16:43.787 INFO:teuthology.orchestra.run.smithi047.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:16:43.820 INFO:teuthology.orchestra.run.smithi047.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:16:43.870 INFO:teuthology.orchestra.run.smithi047.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T18:16:43.937 INFO:teuthology.orchestra.run.smithi047.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:16:43.971 INFO:teuthology.orchestra.run.smithi047.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:16:44.004 INFO:teuthology.orchestra.run.smithi047.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:16:44.038 INFO:teuthology.orchestra.run.smithi047.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:16:44.171 INFO:teuthology.orchestra.run.smithi047.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 938 kB/s | 125 kB 00:00 2024-03-25T18:16:44.439 INFO:teuthology.orchestra.run.smithi047.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-03-25T18:16:44.656 INFO:teuthology.orchestra.run.smithi047.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-25T18:16:44.714 INFO:teuthology.orchestra.run.smithi047.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 745 kB/s | 43 kB 00:00 2024-03-25T18:16:44.725 INFO:teuthology.orchestra.run.smithi142.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 371 kB/s | 4.7 MB 00:12 2024-03-25T18:16:44.856 INFO:teuthology.orchestra.run.smithi047.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T18:16:44.990 INFO:teuthology.orchestra.run.smithi047.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-25T18:16:45.032 INFO:teuthology.orchestra.run.smithi047.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-25T18:16:45.074 INFO:teuthology.orchestra.run.smithi047.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-03-25T18:16:45.116 INFO:teuthology.orchestra.run.smithi047.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 555 kB/s | 23 kB 00:00 2024-03-25T18:16:45.158 INFO:teuthology.orchestra.run.smithi047.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-25T18:16:45.200 INFO:teuthology.orchestra.run.smithi047.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 473 kB/s | 20 kB 00:00 2024-03-25T18:16:45.242 INFO:teuthology.orchestra.run.smithi047.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-25T18:16:45.300 INFO:teuthology.orchestra.run.smithi047.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:16:45.368 INFO:teuthology.orchestra.run.smithi047.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T18:16:45.418 INFO:teuthology.orchestra.run.smithi047.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 330 kB/s | 16 kB 00:00 2024-03-25T18:16:45.468 INFO:teuthology.orchestra.run.smithi047.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T18:16:45.518 INFO:teuthology.orchestra.run.smithi047.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-03-25T18:16:45.577 INFO:teuthology.orchestra.run.smithi047.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-03-25T18:16:45.652 INFO:teuthology.orchestra.run.smithi047.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T18:16:45.702 INFO:teuthology.orchestra.run.smithi047.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-03-25T18:16:48.869 INFO:teuthology.orchestra.run.smithi047.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 505 kB/s | 4.7 MB 00:09 2024-03-25T18:17:13.609 INFO:teuthology.orchestra.run.smithi047.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 534 kB/s | 17 MB 00:32 2024-03-25T18:17:13.612 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:13.612 INFO:teuthology.orchestra.run.smithi047.stdout:Total 863 kB/s | 30 MB 00:35 2024-03-25T18:17:13.636 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:17:13.681 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:17:13.682 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:17:14.097 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:17:14.098 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:17:15.555 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:17:15.619 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:17:15.656 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:17:15.688 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:17:15.763 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:17:15.804 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:17:15.837 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:17:15.924 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:17:16.118 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:17:16.166 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:17:16.240 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:17:16.291 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:17:16.329 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:17:16.387 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:17:16.451 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:17:16.641 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:17:16.691 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:17:16.730 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:17:16.764 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:17:16.794 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:17:16.875 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:17:16.913 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:17:16.946 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:17:16.987 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:17:17.031 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:17:17.065 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:17:17.150 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:17:17.205 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:17:17.279 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:17:17.419 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:17:17.546 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:17:17.711 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:17:17.734 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:17:17.759 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:17:17.759 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:17.759 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:17:17.759 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:17:17.759 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:17:17.759 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:19.276 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:17:19.303 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:17:19.303 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:19.303 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:17:19.304 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:17:19.304 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:17:19.304 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:19.779 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:17:19.810 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:17:19.810 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:19.810 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:17:19.810 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:17:19.811 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:17:19.811 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:20.038 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:17:20.061 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:17:20.061 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:20.061 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:17:20.061 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:17:20.061 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:17:20.061 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:20.121 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:17:20.738 INFO:teuthology.orchestra.run.smithi142.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 374 kB/s | 17 MB 00:45 2024-03-25T18:17:20.742 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:20.743 INFO:teuthology.orchestra.run.smithi142.stdout:Total 609 kB/s | 30 MB 00:49 2024-03-25T18:17:20.765 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:17:20.806 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:17:20.806 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:17:21.235 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:17:21.235 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:17:21.867 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:17:21.868 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:17:21.869 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:17:21.870 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:17:22.549 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:17:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:17:22.654 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:17:22.684 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:17:22.762 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:17:22.808 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:17:22.841 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:17:22.858 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:17:22.859 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:17:22.860 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:22.861 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:22.939 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:17:23.027 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:17:23.069 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:17:23.139 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:17:23.196 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:17:23.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base 2024-03-25T18:17:23.234 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:17:23.284 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:17:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:17:23.521 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:17:23.572 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:17:23.610 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:17:23.645 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:17:23.675 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:17:23.752 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:17:23.785 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:17:23.819 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:17:23.828 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:10 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:23.868 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:17:23.911 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:17:23.928 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:17:23.954 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:17:23.983 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:23.984 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:17:23.984 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:24.053 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:17:24.067 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install cephadm 2024-03-25T18:17:24.102 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:17:24.183 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:17:24.318 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:17:24.446 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:17:24.609 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:17:24.635 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:24.640 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:17:24.664 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:17:24.665 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:24.665 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:17:24.665 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:17:24.665 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:17:24.665 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:24.789 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:17:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:17:24.791 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:24.791 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 747 k 2024-03-25T18:17:24.791 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 754 k 2024-03-25T18:17:24.791 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:17:26.167 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:17:26.196 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:17:26.196 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:26.196 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:17:26.196 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:17:26.196 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:17:26.196 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:26.642 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:17:26.665 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:17:26.665 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:26.665 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:17:26.665 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:17:26.665 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:17:26.666 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:26.820 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 368 kB/s | 747 kB 00:02 2024-03-25T18:17:26.821 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:26.821 INFO:teuthology.orchestra.run.smithi047.stdout:Total 368 kB/s | 747 kB 00:02 2024-03-25T18:17:26.822 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:17:26.828 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:17:26.828 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:17:26.834 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:17:26.835 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:17:26.899 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:17:26.899 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:26.899 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:17:26.899 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:17:26.899 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:17:26.900 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:26.909 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:17:26.967 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:17:27.392 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:27.486 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:27.862 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:28.131 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:28.131 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:28.132 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:17:28.132 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:28.132 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:28.132 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:28.321 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:17:28.565 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:17:28.566 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:17:28.567 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:17:28.569 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:17:28.895 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:29.050 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:29.050 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:17:29.050 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:17:29.050 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:17:29.050 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 145 k 2024-03-25T18:17:29.051 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 436 k 2024-03-25T18:17:29.052 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:17:29.487 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:17:29.487 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:29.487 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:17:29.487 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:17:29.488 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:17:29.489 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:17:29.490 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:17:29.490 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:17:29.490 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:17:29.490 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:29.490 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:17:29.735 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 213 kB/s | 145 kB 00:00 2024-03-25T18:17:29.735 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:29.736 INFO:teuthology.orchestra.run.smithi047.stdout:Total 212 kB/s | 145 kB 00:00 2024-03-25T18:17:29.736 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:17:29.746 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:17:29.746 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:17:29.804 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:17:29.804 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:17:29.904 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-base 2024-03-25T18:17:29.952 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:17:30.028 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:17:30.052 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:17:30.052 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:30.052 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:17:30.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:30.471 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:17:30.568 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:17:30.620 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:17:30.622 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:17:30.622 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:17:30.678 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install cephadm 2024-03-25T18:17:31.002 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:17:31.002 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:31.002 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:17:31.002 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:31.002 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:31.002 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:31.239 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:17:31.257 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr 2024-03-25T18:17:31.388 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:17:31.389 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:17:31.389 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:17:31.389 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:17:31.389 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 747 k 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 754 k 2024-03-25T18:17:31.390 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:17:31.829 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:31.929 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:17:31.984 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:31.985 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:17:31.985 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:32.066 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:17:32.636 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:32.791 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================== 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================== 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:17:32.793 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================== 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout:Install 12 Packages 2024-03-25T18:17:32.794 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:32.795 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 6.9 M 2024-03-25T18:17:32.795 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 26 M 2024-03-25T18:17:32.795 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:17:33.358 INFO:teuthology.orchestra.run.smithi142.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 380 kB/s | 747 kB 00:01 2024-03-25T18:17:33.358 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:33.359 INFO:teuthology.orchestra.run.smithi142.stdout:Total 380 kB/s | 747 kB 00:01 2024-03-25T18:17:33.359 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:17:33.365 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:17:33.365 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:17:33.371 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:17:33.371 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:17:33.450 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:17:33.489 INFO:teuthology.orchestra.run.smithi047.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 68 kB/s | 31 kB 00:00 2024-03-25T18:17:33.598 INFO:teuthology.orchestra.run.smithi047.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T18:17:33.656 INFO:teuthology.orchestra.run.smithi047.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T18:17:33.681 INFO:teuthology.orchestra.run.smithi047.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T18:17:33.882 INFO:teuthology.orchestra.run.smithi047.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-03-25T18:17:33.895 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:34.065 INFO:teuthology.orchestra.run.smithi047.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-25T18:17:34.091 INFO:teuthology.orchestra.run.smithi047.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 93 kB/s | 19 kB 00:00 2024-03-25T18:17:34.232 INFO:teuthology.orchestra.run.smithi047.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-25T18:17:34.258 INFO:teuthology.orchestra.run.smithi047.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 865 kB/s | 144 kB 00:00 2024-03-25T18:17:34.300 INFO:teuthology.orchestra.run.smithi047.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 460 kB/s | 31 kB 00:00 2024-03-25T18:17:34.333 INFO:teuthology.orchestra.run.smithi047.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-25T18:17:34.377 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:17:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:34.648 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:17:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:34.648 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:17:34.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:17:35.434 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:17:35.582 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:17:35.583 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:35.584 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 145 k 2024-03-25T18:17:35.584 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 436 k 2024-03-25T18:17:35.584 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:17:36.413 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 175 kB/s | 145 kB 00:00 2024-03-25T18:17:36.413 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:36.414 INFO:teuthology.orchestra.run.smithi142.stdout:Total 175 kB/s | 145 kB 00:00 2024-03-25T18:17:36.414 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:17:36.422 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:17:36.422 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:17:36.475 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:17:36.475 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:17:36.629 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:17:36.697 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:17:36.721 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:17:36.721 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:17:36.721 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:17:36.721 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:36.883 INFO:teuthology.orchestra.run.smithi047.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 473 kB/s | 1.8 MB 00:03 2024-03-25T18:17:36.887 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:36.888 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.7 MB/s | 6.9 MB 00:04 2024-03-25T18:17:36.908 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:17:36.927 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:17:36.927 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:17:37.107 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:17:37.108 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:17:37.626 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:17:37.709 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:17:37.735 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:17:37.735 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:37.735 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:17:37.735 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:37.735 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:37.735 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:17:37.744 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:17:37.791 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:17:37.829 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:17:37.945 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:17:37.999 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr 2024-03-25T18:17:38.076 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:17:38.169 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:17:38.218 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:17:38.246 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:17:38.278 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:17:38.523 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:17:38.553 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:17:38.576 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:17:38.676 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:17:38.728 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:17:38.729 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:17:38.729 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:17:38.789 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:17:39.313 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:17:39.314 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:17:39.350 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:17:39.500 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:17:39.501 INFO:teuthology.orchestra.run.smithi142.stdout:====================================================================================== 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout:====================================================================================== 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:17:39.502 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout:====================================================================================== 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout:Install 12 Packages 2024-03-25T18:17:39.503 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:39.504 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 6.9 M 2024-03-25T18:17:39.504 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 26 M 2024-03-25T18:17:39.504 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:39.705 INFO:teuthology.orchestra.run.smithi047.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:39.706 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:39.944 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 68 kB/s | 31 kB 00:00 2024-03-25T18:17:40.208 INFO:teuthology.orchestra.run.smithi142.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 30 kB/s | 15 kB 00:00 2024-03-25T18:17:40.300 INFO:teuthology.orchestra.run.smithi142.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.6 MB/s | 238 kB 00:00 2024-03-25T18:17:40.359 INFO:teuthology.orchestra.run.smithi142.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T18:17:40.517 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:40.676 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:17:40.677 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout:Install 12 Packages 2024-03-25T18:17:40.678 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:40.679 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 37 M 2024-03-25T18:17:40.679 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 209 M 2024-03-25T18:17:40.679 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:17:40.793 INFO:teuthology.orchestra.run.smithi142.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T18:17:40.859 INFO:teuthology.orchestra.run.smithi142.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-03-25T18:17:41.115 INFO:teuthology.orchestra.run.smithi047.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 268 kB/s | 32 kB 00:00 2024-03-25T18:17:41.135 INFO:teuthology.orchestra.run.smithi142.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-25T18:17:41.174 INFO:teuthology.orchestra.run.smithi047.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 497 kB/s | 17 kB 00:00 2024-03-25T18:17:41.199 INFO:teuthology.orchestra.run.smithi047.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T18:17:41.508 INFO:teuthology.orchestra.run.smithi047.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-25T18:17:41.543 INFO:teuthology.orchestra.run.smithi142.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 3.0 MB/s | 2.0 MB 00:00 2024-03-25T18:17:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 275 kB/s | 144 kB 00:00 2024-03-25T18:17:41.800 INFO:teuthology.orchestra.run.smithi047.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T18:17:41.841 INFO:teuthology.orchestra.run.smithi047.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T18:17:41.869 INFO:teuthology.orchestra.run.smithi142.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 95 kB/s | 31 kB 00:00 2024-03-25T18:17:41.927 INFO:teuthology.orchestra.run.smithi142.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2024-03-25T18:17:42.092 INFO:teuthology.orchestra.run.smithi047.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-03-25T18:17:42.326 INFO:teuthology.orchestra.run.smithi047.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-25T18:17:42.459 INFO:teuthology.orchestra.run.smithi047.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-25T18:17:42.501 INFO:teuthology.orchestra.run.smithi047.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T18:17:42.959 INFO:teuthology.orchestra.run.smithi047.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 491 kB/s | 245 kB 00:00 2024-03-25T18:17:43.603 INFO:teuthology.orchestra.run.smithi142.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 467 kB/s | 1.8 MB 00:03 2024-03-25T18:17:43.608 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:43.608 INFO:teuthology.orchestra.run.smithi142.stdout:Total 1.7 MB/s | 6.9 MB 00:04 2024-03-25T18:17:43.628 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:17:43.648 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:17:43.648 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:17:43.828 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:17:43.828 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:17:44.436 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:17:44.472 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:17:44.518 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:17:44.548 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:17:44.648 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:17:44.782 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:17:44.956 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:17:44.988 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:17:45.231 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:17:45.263 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:17:46.018 INFO:teuthology.orchestra.run.smithi047.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.5 MB/s | 7.4 MB 00:05 2024-03-25T18:17:46.020 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:46.021 INFO:teuthology.orchestra.run.smithi047.stdout:Total 6.9 MB/s | 37 MB 00:05 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:17:46.022 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:17:46.023 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:17:46.023 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:17:46.023 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:17:46.023 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:17:46.023 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:17:46.065 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:17:46.091 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:17:46.091 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:17:46.380 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:17:46.380 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:46.380 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:17:46.380 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:17:46.380 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:46.381 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:17:46.463 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:17:46.464 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:17:46.626 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:17:47.185 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:17:47.187 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:32 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:17:47.250 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:17:47.281 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:17:47.339 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:17:47.340 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================================ 2024-03-25T18:17:47.340 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:17:47.340 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================================ 2024-03-25T18:17:47.340 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:17:47.340 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:17:47.340 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:17:47.341 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:17:47.342 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:47.342 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:17:47.342 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================================ 2024-03-25T18:17:47.342 INFO:teuthology.orchestra.run.smithi142.stdout:Install 12 Packages 2024-03-25T18:17:47.342 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:17:47.343 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 37 M 2024-03-25T18:17:47.343 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 209 M 2024-03-25T18:17:47.343 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:17:47.380 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:17:47.417 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:17:47.605 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:17:47.633 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:17:47.713 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:17:47.868 INFO:teuthology.orchestra.run.smithi142.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 228 kB/s | 32 kB 00:00 2024-03-25T18:17:47.926 INFO:teuthology.orchestra.run.smithi142.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 493 kB/s | 17 kB 00:00 2024-03-25T18:17:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:17:48.119 INFO:teuthology.orchestra.run.smithi142.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:17:48.252 INFO:teuthology.orchestra.run.smithi142.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:17:48.319 INFO:teuthology.orchestra.run.smithi142.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T18:17:48.344 INFO:teuthology.orchestra.run.smithi142.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T18:17:48.400 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:17:48.517 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:17:48.653 INFO:teuthology.orchestra.run.smithi142.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-25T18:17:48.870 INFO:teuthology.orchestra.run.smithi142.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 903 kB/s | 195 kB 00:00 2024-03-25T18:17:48.936 INFO:teuthology.orchestra.run.smithi142.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T18:17:49.020 INFO:teuthology.orchestra.run.smithi142.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-25T18:17:49.847 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:17:50.926 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:17:50.957 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:17:53.402 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:17:53.403 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:17:53.852 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:53.853 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:54.230 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:17:54.814 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:17:54.972 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:17:54.973 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:54.974 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:17:54.975 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:17:54.975 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2024-03-25T18:17:54.975 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:54.975 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 2.0 M 2024-03-25T18:17:54.975 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 26 M 2024-03-25T18:17:54.975 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:17:55.151 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 18 kB 00:00 2024-03-25T18:17:55.218 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T18:17:55.243 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 110 kB/s | 24 kB 00:00 2024-03-25T18:17:55.268 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-25T18:17:55.302 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T18:17:55.327 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 171 kB/s | 52 kB 00:00 2024-03-25T18:17:55.353 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 511 kB/s | 43 kB 00:00 2024-03-25T18:17:55.470 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 129 kB/s | 15 kB 00:00 2024-03-25T18:17:55.495 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-25T18:17:55.595 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 838 kB/s | 203 kB 00:00 2024-03-25T18:17:55.620 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-25T18:17:55.679 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-25T18:17:55.746 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-25T18:17:55.748 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:55.748 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-25T18:17:55.748 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:17:55.763 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:17:55.763 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:17:55.864 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:17:55.864 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:17:56.380 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:17:56.558 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:17:56.615 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:17:56.670 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:17:56.706 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:17:56.748 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:17:56.845 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:17:56.953 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:17:57.016 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:17:57.775 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:17:57.822 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:17:57.855 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:17:57.905 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:17:57.933 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:17:58.700 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:17:58.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:17:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:17:59.060 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:17:59.285 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:17:59.862 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:00.018 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:18:00.019 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-03-25T18:18:00.020 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:00.021 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 6.8 M 2024-03-25T18:18:00.021 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 33 M 2024-03-25T18:18:00.021 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:00.515 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T18:18:00.623 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:18:00.715 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 636 kB/s | 58 kB 00:00 2024-03-25T18:18:00.765 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-25T18:18:00.932 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T18:18:01.324 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 141 kB/s | 138 kB 00:00 2024-03-25T18:18:01.326 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:01.326 INFO:teuthology.orchestra.run.smithi047.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2024-03-25T18:18:01.341 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:01.353 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:01.353 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:01.449 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:01.451 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:01.749 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:01.914 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:18:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:18:02.517 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:18:02.607 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:18:02.692 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:18:02.730 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:18:02.810 INFO:teuthology.orchestra.run.smithi142.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 502 kB/s | 7.4 MB 00:15 2024-03-25T18:18:02.813 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:02.813 INFO:teuthology.orchestra.run.smithi142.stdout:Total 2.4 MB/s | 37 MB 00:15 2024-03-25T18:18:02.855 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:02.881 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:02.882 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:03.248 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:03.249 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:03.540 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:18:03.540 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:18:03.541 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:18:03.541 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:18:03.541 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:18:03.541 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:03.831 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:04.025 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:18:04.048 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse 2024-03-25T18:18:04.055 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:18:04.165 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:18:04.209 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:18:04.386 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:18:04.417 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:18:04.493 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:18:04.622 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:04.780 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:18:04.781 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 964 k 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.8 M 2024-03-25T18:18:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:04.901 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 662 kB/s | 79 kB 00:00 2024-03-25T18:18:05.145 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:18:05.262 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:18:06.492 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:18:07.343 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 346 kB/s | 885 kB 00:02 2024-03-25T18:18:07.344 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:07.345 INFO:teuthology.orchestra.run.smithi047.stdout:Total 376 kB/s | 964 kB 00:02 2024-03-25T18:18:07.345 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:07.370 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:07.370 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:07.450 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:07.450 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:07.531 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:18:07.574 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:18:07.637 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:07.781 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:18:07.861 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:18:08.539 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:18:08.539 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:18:08.830 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:18:08.831 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:08.831 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:08.831 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:18:08.831 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:08.831 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:09.106 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-volume 2024-03-25T18:18:09.682 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:09.840 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:09.840 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:09.840 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 332 k 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.5 M 2024-03-25T18:18:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:09.939 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 706 kB/s | 69 kB 00:00 2024-03-25T18:18:10.236 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:18:10.236 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:18:10.236 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:18:10.236 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:18:10.236 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:18:10.237 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:18:10.712 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:10.713 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:11.014 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 225 kB/s | 264 kB 00:01 2024-03-25T18:18:11.016 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:11.016 INFO:teuthology.orchestra.run.smithi047.stdout:Total 283 kB/s | 332 kB 00:01 2024-03-25T18:18:11.016 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:11.027 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:11.027 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:11.076 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:11.076 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:11.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:18:11.233 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:11.431 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:18:11.467 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:18:11.482 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:18:11.482 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:18:11.482 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:18:11.482 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:11.677 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:11.829 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:11.830 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:18:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:18:11.830 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:18:11.830 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:18:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout:Install 13 Packages 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 2.0 M 2024-03-25T18:18:11.832 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 26 M 2024-03-25T18:18:11.833 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:11.975 INFO:teuthology.orchestra.run.smithi142.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 250 kB/s | 24 kB 00:00 2024-03-25T18:18:12.017 INFO:teuthology.orchestra.run.smithi142.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-25T18:18:12.059 INFO:teuthology.orchestra.run.smithi142.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T18:18:12.084 INFO:teuthology.orchestra.run.smithi142.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:18:12.109 INFO:teuthology.orchestra.run.smithi142.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:18:12.134 INFO:teuthology.orchestra.run.smithi142.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 869 kB/s | 43 kB 00:00 2024-03-25T18:18:12.200 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-25T18:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-03-25T18:18:12.418 INFO:teuthology.orchestra.run.smithi142.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 96 kB/s | 52 kB 00:00 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi142.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:12.519 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:12.594 INFO:teuthology.orchestra.run.smithi142.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-25T18:18:12.644 INFO:teuthology.orchestra.run.smithi142.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-25T18:18:12.670 INFO:teuthology.orchestra.run.smithi142.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 235 kB/s | 59 kB 00:00 2024-03-25T18:18:12.674 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:12.674 INFO:teuthology.orchestra.run.smithi142.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-25T18:18:12.674 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:12.687 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:12.687 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:12.828 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel 2024-03-25T18:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:13.408 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:13.466 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:18:13.528 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:18:13.563 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:13.564 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:18:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:13.565 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 127 k 2024-03-25T18:18:13.565 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 456 k 2024-03-25T18:18:13.565 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:13.592 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:18:13.630 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:18:13.671 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:18:13.763 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:18:13.865 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:18:13.914 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:18:14.588 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 124 kB/s | 127 kB 00:01 2024-03-25T18:18:14.588 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:14.589 INFO:teuthology.orchestra.run.smithi047.stdout:Total 124 kB/s | 127 kB 00:01 2024-03-25T18:18:14.589 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:14.595 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:14.595 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:14.640 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:18:14.659 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:14.659 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:14.695 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:18:14.729 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:18:14.777 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:18:14.799 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:18:14.821 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:14.930 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:15.484 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:15.580 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:18:15.580 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:18:15.580 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:18:15.580 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:18:15.581 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:18:15.769 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:15.770 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:15.770 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:15.770 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:15.770 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:15.770 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:18:15.938 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:15.939 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:15.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2 2024-03-25T18:18:16.193 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:18:16.554 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:16.657 INFO:teuthology.orchestra.run.smithi047.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:16.713 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:16.714 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:16.714 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:16.762 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:16.802 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs-devel 2024-03-25T18:18:16.915 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:16.916 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:18:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:18:16.916 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:18:16.916 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout:Install 6 Packages 2024-03-25T18:18:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:16.918 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 6.8 M 2024-03-25T18:18:16.918 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 33 M 2024-03-25T18:18:16.918 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:17.376 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:18:17.535 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:17.535 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:17.535 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:18:17.536 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:17.537 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 31 k 2024-03-25T18:18:17.537 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 140 k 2024-03-25T18:18:17.537 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:17.930 INFO:teuthology.orchestra.run.smithi142.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 356 kB/s | 248 kB 00:00 2024-03-25T18:18:17.955 INFO:teuthology.orchestra.run.smithi142.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 191 kB/s | 138 kB 00:00 2024-03-25T18:18:18.066 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:18:18.066 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:18.067 INFO:teuthology.orchestra.run.smithi047.stdout:Total 59 kB/s | 31 kB 00:00 2024-03-25T18:18:18.067 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:18.071 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:18.071 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:18.095 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:18.096 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:18.227 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:18.325 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:18.525 INFO:teuthology.orchestra.run.smithi142.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 97 kB/s | 58 kB 00:00 2024-03-25T18:18:18.556 INFO:teuthology.orchestra.run.smithi142.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 85 kB/s | 51 kB 00:00 2024-03-25T18:18:18.606 INFO:teuthology.orchestra.run.smithi142.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 502 kB/s | 548 kB 00:01 2024-03-25T18:18:18.609 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:18.609 INFO:teuthology.orchestra.run.smithi142.stdout:Total 4.0 MB/s | 6.8 MB 00:01 2024-03-25T18:18:18.623 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:18.633 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:18.633 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:18.716 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:18.723 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:18.723 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:18.985 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:18.985 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:18.985 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:18.986 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:18.986 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:18.986 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:19.022 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:19.147 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2 2024-03-25T18:18:19.188 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:18:19.240 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:18:19.718 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:19.756 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:18:19.816 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:19.834 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:18:19.871 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:19.873 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:19.873 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:19.904 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:18:19.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1 2024-03-25T18:18:19.945 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:18:20.492 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:20.592 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:20.646 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:20.648 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:20.648 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:20.696 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados 2024-03-25T18:18:20.760 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:18:20.760 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:18:20.760 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:18:20.760 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:18:20.760 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:18:20.761 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:18:21.072 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:18:21.072 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:21.073 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:21.271 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:21.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-fuse 2024-03-25T18:18:21.374 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:21.429 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:21.430 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:21.430 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:21.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw 2024-03-25T18:18:21.875 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:22.025 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:22.026 INFO:teuthology.orchestra.run.smithi142.stdout:Install 2 Packages 2024-03-25T18:18:22.027 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:22.027 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 964 k 2024-03-25T18:18:22.027 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 2.8 M 2024-03-25T18:18:22.027 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:22.066 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:22.140 INFO:teuthology.orchestra.run.smithi142.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 697 kB/s | 79 kB 00:00 2024-03-25T18:18:22.170 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:22.224 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:22.225 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:22.226 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:22.268 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs 2024-03-25T18:18:22.850 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:22.949 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:23.005 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:23.006 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:23.006 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:23.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd 2024-03-25T18:18:23.486 INFO:teuthology.orchestra.run.smithi142.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 607 kB/s | 885 kB 00:01 2024-03-25T18:18:23.487 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:23.487 INFO:teuthology.orchestra.run.smithi142.stdout:Total 660 kB/s | 964 kB 00:01 2024-03-25T18:18:23.488 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:23.510 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:23.511 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:23.587 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:23.587 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:23.620 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:23.720 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:23.774 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:23.776 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T18:18:23.776 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:23.819 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse 2024-03-25T18:18:23.908 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:18:23.974 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:18:24.392 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:24.547 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:18:24.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:24.549 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 85 k 2024-03-25T18:18:24.549 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 231 k 2024-03-25T18:18:24.549 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:24.626 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:18:24.626 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:18:24.921 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:18:24.921 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:24.921 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:24.921 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:18:24.921 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:24.921 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:25.268 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-volume 2024-03-25T18:18:25.278 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 117 kB/s | 85 kB 00:00 2024-03-25T18:18:25.278 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:25.279 INFO:teuthology.orchestra.run.smithi047.stdout:Total 117 kB/s | 85 kB 00:00 2024-03-25T18:18:25.279 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:25.287 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:25.287 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:25.348 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:25.348 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:25.493 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:25.604 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:25.843 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:25.998 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout:Install 2 Packages 2024-03-25T18:18:25.999 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:26.000 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 332 k 2024-03-25T18:18:26.000 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 1.5 M 2024-03-25T18:18:26.000 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:26.009 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:26.112 INFO:teuthology.orchestra.run.smithi142.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 619 kB/s | 69 kB 00:00 2024-03-25T18:18:26.286 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:26.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:26.286 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:26.286 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:26.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:26.286 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:26.503 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror 2024-03-25T18:18:27.077 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:27.232 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.1 M 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 13 M 2024-03-25T18:18:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:27.662 INFO:teuthology.orchestra.run.smithi142.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 159 kB/s | 264 kB 00:01 2024-03-25T18:18:27.663 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:27.663 INFO:teuthology.orchestra.run.smithi142.stdout:Total 200 kB/s | 332 kB 00:01 2024-03-25T18:18:27.664 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:27.674 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:27.674 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:27.720 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:27.720 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:27.877 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:28.063 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:18:28.097 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:18:28.113 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:18:28.113 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:18:28.113 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:18:28.113 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:28.847 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:18:29.236 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:18:29.236 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:29.236 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:29.236 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:18:29.237 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:18:29.237 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:29.237 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:29.586 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librados-devel 2024-03-25T18:18:30.159 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 127 k 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 456 k 2024-03-25T18:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:31.951 INFO:teuthology.orchestra.run.smithi142.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 77 kB/s | 127 kB 00:01 2024-03-25T18:18:31.951 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:31.952 INFO:teuthology.orchestra.run.smithi142.stdout:Total 77 kB/s | 127 kB 00:01 2024-03-25T18:18:31.952 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:31.959 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:31.959 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:32.021 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:32.021 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:32.179 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:32.284 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:32.728 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:33.020 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:33.020 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:33.020 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:33.020 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:33.020 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:33.020 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:33.252 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs2 2024-03-25T18:18:33.827 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:33.928 INFO:teuthology.orchestra.run.smithi142.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:33.981 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:33.983 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:33.983 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:34.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs-devel 2024-03-25T18:18:34.455 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 438 kB/s | 3.1 MB 00:07 2024-03-25T18:18:34.457 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:34.457 INFO:teuthology.orchestra.run.smithi047.stdout:Total 438 kB/s | 3.1 MB 00:07 2024-03-25T18:18:34.457 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:34.469 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:34.469 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:34.548 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:34.549 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:34.612 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:34.761 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:18:34.762 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:34.763 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 31 k 2024-03-25T18:18:34.763 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 140 k 2024-03-25T18:18:34.763 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:35.005 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:35.065 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:35.089 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:35.089 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:18:35.089 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:18:35.089 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:18:35.089 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:18:35.089 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:35.287 INFO:teuthology.orchestra.run.smithi142.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:18:35.288 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:35.288 INFO:teuthology.orchestra.run.smithi142.stdout:Total 59 kB/s | 31 kB 00:00 2024-03-25T18:18:35.288 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:35.292 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:35.292 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:35.320 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:35.320 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:35.437 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:35.562 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:35.958 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:36.184 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:36.185 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:36.185 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:36.185 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:36.185 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:36.185 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:36.220 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:36.220 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:36.221 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:36.221 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:36.221 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:36.221 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:36.395 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librados2 2024-03-25T18:18:36.414 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd 2024-03-25T18:18:36.960 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:36.990 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:37.059 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:37.111 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:37.112 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:37.112 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:37.146 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:37.147 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:18:37.148 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:37.148 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 171 k 2024-03-25T18:18:37.148 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 494 k 2024-03-25T18:18:37.148 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:37.174 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librbd1 2024-03-25T18:18:37.745 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:37.850 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:37.903 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:37.904 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:37.904 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:37.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rados 2024-03-25T18:18:38.051 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 190 kB/s | 171 kB 00:00 2024-03-25T18:18:38.051 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:38.052 INFO:teuthology.orchestra.run.smithi047.stdout:Total 190 kB/s | 171 kB 00:00 2024-03-25T18:18:38.052 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:18:38.062 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:18:38.062 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:18:38.126 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:18:38.126 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:18:38.284 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:18:38.394 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:38.548 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:38.650 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:38.703 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:38.705 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:38.705 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:38.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rgw 2024-03-25T18:18:38.819 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:39.068 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:39.068 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:39.068 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:18:39.068 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:39.069 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:39.069 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:18:39.330 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:39.336 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:18:39.427 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:39.478 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:39.479 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:39.479 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:39.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-cephfs 2024-03-25T18:18:39.912 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:18:40.068 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-25T18:18:40.069 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:18:40.070 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 91 M 2024-03-25T18:18:40.070 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 498 M 2024-03-25T18:18:40.070 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:18:40.099 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:40.196 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:40.247 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:40.248 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:40.248 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:40.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rbd 2024-03-25T18:18:40.863 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:40.959 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:41.010 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:41.012 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T18:18:41.012 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:41.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-fuse 2024-03-25T18:18:41.630 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:41.777 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:41.778 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:41.778 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-03-25T18:18:41.778 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:41.778 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:41.778 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:18:41.778 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 85 k 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 231 k 2024-03-25T18:18:41.779 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:42.582 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 106 kB/s | 85 kB 00:00 2024-03-25T18:18:42.583 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:42.583 INFO:teuthology.orchestra.run.smithi142.stdout:Total 106 kB/s | 85 kB 00:00 2024-03-25T18:18:42.583 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:42.591 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:42.591 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:42.648 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:42.648 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:42.789 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:42.889 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:43.329 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:43.623 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:43.623 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:43.623 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:43.623 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:43.623 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:43.623 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:43.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-mirror 2024-03-25T18:18:44.456 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:44.615 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:44.616 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:18:44.616 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:44.616 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 3.1 M 2024-03-25T18:18:44.616 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 13 M 2024-03-25T18:18:44.616 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:53.978 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 338 kB/s | 3.1 MB 00:09 2024-03-25T18:18:53.979 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:53.979 INFO:teuthology.orchestra.run.smithi142.stdout:Total 338 kB/s | 3.1 MB 00:09 2024-03-25T18:18:53.980 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:53.991 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:53.991 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:54.067 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:54.067 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:54.498 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:54.562 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:54.586 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:54.586 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:18:54.586 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:18:54.586 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:18:54.586 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:18:54.587 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:55.687 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:55.688 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:55.688 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:55.688 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:55.688 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:55.688 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:55.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-nbd 2024-03-25T18:18:56.540 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:56.694 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:56.694 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:56.694 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2024-03-25T18:18:56.694 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:56.694 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:56.695 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 171 k 2024-03-25T18:18:56.696 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 494 k 2024-03-25T18:18:56.696 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:18:57.365 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 256 kB/s | 171 kB 00:00 2024-03-25T18:18:57.365 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:57.366 INFO:teuthology.orchestra.run.smithi142.stdout:Total 256 kB/s | 171 kB 00:00 2024-03-25T18:18:57.366 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:18:57.374 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:18:57.374 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:18:57.430 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:18:57.430 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:18:57.572 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:18:57.698 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:58.125 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:58.429 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:18:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:58.429 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:18:58.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:18:59.276 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:18:59.429 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:Install 2 Packages 2024-03-25T18:18:59.431 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:18:59.432 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 91 M 2024-03-25T18:18:59.432 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 498 M 2024-03-25T18:18:59.432 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:19:31.014 INFO:teuthology.orchestra.run.smithi142.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 728 kB/s | 22 MB 00:31 2024-03-25T18:19:46.823 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 344 kB/s | 22 MB 01:06 2024-03-25T18:21:26.871 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 418 kB/s | 68 MB 02:46 2024-03-25T18:21:26.872 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:26.872 INFO:teuthology.orchestra.run.smithi047.stdout:Total 556 kB/s | 91 MB 02:46 2024-03-25T18:21:26.872 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:21:26.877 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:21:26.877 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:21:27.492 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:21:27.492 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:21:30.768 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:21:38.276 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:21:38.350 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:21:41.750 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:21:41.750 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:21:42.013 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:21:42.013 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:21:42.014 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:21:42.014 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:42.014 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:42.014 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:21:42.014 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:21:42.224 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:21:42.798 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:29 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:21:42.951 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 416 M 2024-03-25T18:21:42.952 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.7 G 2024-03-25T18:21:42.953 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:21:58.036 INFO:teuthology.orchestra.run.smithi142.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 391 kB/s | 68 MB 02:58 2024-03-25T18:21:58.037 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:58.037 INFO:teuthology.orchestra.run.smithi142.stdout:Total 519 kB/s | 91 MB 02:58 2024-03-25T18:21:58.038 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:21:58.042 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:21:58.042 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:21:58.635 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:21:58.635 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:22:01.773 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:22:08.986 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:22:09.127 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:22:12.477 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:22:12.477 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:22:12.742 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:22:12.742 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:22:12.742 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:22:12.742 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:12.743 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:12.743 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:22:12.743 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:22:12.960 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:22:13.525 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:08:58 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:22:13.673 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:22:13.673 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:22:13.673 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:22:13.673 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:22:13.673 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:22:13.673 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 416 M 2024-03-25T18:22:13.674 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 1.7 G 2024-03-25T18:22:13.675 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:33:09.773 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 649 kB/s | 416 MB 10:56 2024-03-25T18:33:09.773 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:09.773 INFO:teuthology.orchestra.run.smithi142.stdout:Total 649 kB/s | 416 MB 10:56 2024-03-25T18:33:09.773 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:33:09.776 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:33:09.776 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:33:12.244 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:33:12.245 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:33:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:33:54.498 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:34:02.223 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:34:02.475 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:34:02.475 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:02.476 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:34:02.476 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:02.476 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:02.476 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:34:02.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:34:03.292 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:20:48 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:34:03.441 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================== 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:03.442 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 1.4 M 2024-03-25T18:34:03.443 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 6.4 M 2024-03-25T18:34:03.443 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:34:08.510 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 288 kB/s | 1.4 MB 00:05 2024-03-25T18:34:08.510 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:08.511 INFO:teuthology.orchestra.run.smithi142.stdout:Total 288 kB/s | 1.4 MB 00:05 2024-03-25T18:34:08.511 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:34:08.515 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:34:08.515 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:34:08.560 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:34:08.561 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:34:08.814 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:34:08.926 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:34:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:34:09.647 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:34:09.647 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:09.647 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:34:09.647 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:09.647 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:09.647 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:34:09.828 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:34:10.389 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:20:55 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:34:10.539 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:34:10.540 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:34:10.541 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 176 M 2024-03-25T18:34:10.541 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 753 M 2024-03-25T18:34:10.541 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:37:04.176 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 463 kB/s | 416 MB 15:21 2024-03-25T18:37:04.177 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:04.177 INFO:teuthology.orchestra.run.smithi047.stdout:Total 463 kB/s | 416 MB 15:21 2024-03-25T18:37:04.177 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:37:04.180 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:37:04.180 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:37:06.766 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:37:06.766 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:37:51.493 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:37:51.582 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:37:58.312 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:37:58.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:37:58.576 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:37:58.576 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:37:58.576 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:58.576 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:37:58.576 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:37:58.818 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:37:59.392 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:24:46 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:37:59.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:37:59.549 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.4 M 2024-03-25T18:37:59.549 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 6.4 M 2024-03-25T18:37:59.549 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:38:04.274 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 309 kB/s | 1.4 MB 00:04 2024-03-25T18:38:04.275 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:04.275 INFO:teuthology.orchestra.run.smithi047.stdout:Total 309 kB/s | 1.4 MB 00:04 2024-03-25T18:38:04.275 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:38:04.279 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:38:04.280 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:38:04.326 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:38:04.327 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:38:04.597 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:38:04.700 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:38:05.221 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:38:05.507 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:38:05.507 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:38:05.507 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:38:05.507 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:05.507 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:38:05.507 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:38:05.697 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:38:06.264 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:24:53 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:38:06.418 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:38:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:38:06.419 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:38:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:38:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 176 M 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 753 M 2024-03-25T18:38:06.420 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T18:42:40.432 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 353 kB/s | 176 MB 08:29 2024-03-25T18:42:40.433 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:42:40.433 INFO:teuthology.orchestra.run.smithi142.stdout:Total 353 kB/s | 176 MB 08:29 2024-03-25T18:42:40.433 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T18:42:40.436 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T18:42:40.436 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T18:42:41.496 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T18:42:41.496 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T18:42:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T18:42:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:43:04.394 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:43:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:43:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:43:04.662 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T18:43:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:43:04.662 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T18:43:04.867 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:43:05.437 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:29:50 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T18:43:05.585 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T18:43:05.586 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:43:05.586 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 672 M 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 3.1 G 2024-03-25T18:43:05.587 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T18:47:03.419 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 335 kB/s | 176 MB 08:56 2024-03-25T18:47:03.420 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:03.420 INFO:teuthology.orchestra.run.smithi047.stdout:Total 335 kB/s | 176 MB 08:56 2024-03-25T18:47:03.420 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T18:47:03.423 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T18:47:03.423 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T18:47:04.533 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T18:47:04.533 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T18:47:23.557 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T18:47:23.677 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:47:28.061 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:47:28.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:47:28.362 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:47:28.362 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T18:47:28.362 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:28.362 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:47:28.362 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T18:47:28.563 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:47:29.141 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:34:16 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T18:47:29.295 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T18:47:29.296 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:47:29.296 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T18:47:29.296 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:47:29.296 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T18:47:29.296 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:47:29.296 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 672 M 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 3.1 G 2024-03-25T18:47:29.297 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:04:58.018 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 524 kB/s | 672 MB 21:52 2024-03-25T19:04:58.019 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:58.019 INFO:teuthology.orchestra.run.smithi142.stdout:Total 524 kB/s | 672 MB 21:52 2024-03-25T19:04:58.019 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:04:58.024 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:04:58.024 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:05:01.997 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:05:01.998 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:06:12.920 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:06:13.034 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:06:22.074 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:06:22.337 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:06:22.337 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:06:22.337 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:06:22.337 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:22.337 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:06:22.337 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:06:22.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:06:23.183 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:53:08 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:06:23.279 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:06:23.331 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:06:23.332 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2024-03-25T19:06:23.332 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:06:23.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:53:08 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:06:24.098 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:06:24.099 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:06:24.100 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 20 M 2024-03-25T19:06:24.100 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 103 M 2024-03-25T19:06:24.100 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:07:21.845 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 349 kB/s | 20 MB 00:57 2024-03-25T19:07:21.846 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:21.846 INFO:teuthology.orchestra.run.smithi142.stdout:Total 349 kB/s | 20 MB 00:57 2024-03-25T19:07:21.846 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:07:21.850 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:07:21.850 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:07:22.000 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:07:22.000 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:07:24.234 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:07:24.519 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:25.753 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:25.753 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:07:25.753 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:07:25.753 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:25.753 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:07:25.754 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:07:25.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:07:26.507 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:54:11 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:07:26.654 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:07:26.654 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:07:26.654 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:07:26.654 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:07:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:07:26.656 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 48 M 2024-03-25T19:07:26.656 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 173 M 2024-03-25T19:07:26.656 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:08:31.005 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 759 kB/s | 48 MB 01:04 2024-03-25T19:08:31.006 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:08:31.006 INFO:teuthology.orchestra.run.smithi142.stdout:Total 759 kB/s | 48 MB 01:04 2024-03-25T19:08:31.006 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:08:31.010 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:08:31.010 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:08:31.320 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:08:31.321 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:08:36.046 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:08:36.186 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:08:37.502 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:08:37.774 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:08:37.774 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:08:37.774 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:08:37.774 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:08:37.774 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:08:37.774 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:08:37.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:08:38.513 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:55:23 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:08:38.660 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:08:38.661 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:08:38.662 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 99 M 2024-03-25T19:08:38.662 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 449 M 2024-03-25T19:08:38.662 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:12:34.359 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 428 kB/s | 99 MB 03:55 2024-03-25T19:12:34.359 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:34.359 INFO:teuthology.orchestra.run.smithi142.stdout:Total 428 kB/s | 99 MB 03:55 2024-03-25T19:12:34.360 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:12:34.362 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:12:34.362 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:12:34.970 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:12:34.970 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:12:45.212 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:12:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:48.098 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:12:48.348 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:12:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:12:48.348 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:12:48.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:12:49.128 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:59:34 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:12:49.284 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:12:49.285 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:12:49.286 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:12:49.286 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:12:49.286 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 287 M 2024-03-25T19:12:49.286 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 1.2 G 2024-03-25T19:12:49.286 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:19:33.668 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 358 kB/s | 672 MB 32:04 2024-03-25T19:19:33.668 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:33.668 INFO:teuthology.orchestra.run.smithi047.stdout:Total 358 kB/s | 672 MB 32:04 2024-03-25T19:19:33.669 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:19:33.673 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:19:33.673 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:19:37.829 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:19:37.830 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:20:52.711 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:20:52.822 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:00.328 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:00.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:00.611 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:00.611 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:21:00.611 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:00.611 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:00.611 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:21:00.920 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:21:01.495 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:07:48 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:21:01.594 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:21:01.649 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:21:01.650 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-25T19:21:01.650 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:21:01.724 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:21:02.302 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:07:49 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:21:02.457 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:02.458 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 20 M 2024-03-25T19:21:02.459 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 103 M 2024-03-25T19:21:02.459 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:21:49.001 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 433 kB/s | 20 MB 00:46 2024-03-25T19:21:49.002 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:49.003 INFO:teuthology.orchestra.run.smithi047.stdout:Total 433 kB/s | 20 MB 00:46 2024-03-25T19:21:49.003 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:21:49.007 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:21:49.007 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:21:49.163 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:21:49.164 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:21:51.512 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:21:51.610 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:52.556 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:52.833 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:52.834 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:52.834 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:21:52.834 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:52.834 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:52.834 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:21:53.033 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:21:53.607 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:08:40 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:21:53.763 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:53.764 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:21:53.765 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:21:53.765 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:21:53.765 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:21:53.765 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 48 M 2024-03-25T19:21:53.765 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 173 M 2024-03-25T19:21:53.765 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:23:16.779 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 589 kB/s | 48 MB 01:23 2024-03-25T19:23:16.780 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:16.780 INFO:teuthology.orchestra.run.smithi047.stdout:Total 589 kB/s | 48 MB 01:23 2024-03-25T19:23:16.780 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:23:16.783 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:23:16.784 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:23:17.111 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:23:17.111 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:23:18.094 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 467 kB/s | 287 MB 10:28 2024-03-25T19:23:18.094 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:18.095 INFO:teuthology.orchestra.run.smithi142.stdout:Total 467 kB/s | 287 MB 10:28 2024-03-25T19:23:18.095 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:23:18.098 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:23:18.099 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:23:19.807 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:23:19.808 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:23:22.137 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:23:22.238 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:23.559 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:23.841 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:23.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:23:23.842 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:23:23.842 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:23.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:23:23.842 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:23:24.027 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:23:24.600 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:10:11 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:23:24.756 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:23:24.756 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 99 M 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 449 M 2024-03-25T19:23:24.757 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:23:49.326 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:23:49.423 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:50.615 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:50.929 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:50.929 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:23:50.929 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:23:50.929 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:50.929 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:23:50.929 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:23:51.147 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:23:51.707 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:10:36 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:23:51.857 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:23:51.858 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:23:51.858 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 5.9 M 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 28 M 2024-03-25T19:23:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:24:08.858 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 358 kB/s | 5.9 MB 00:16 2024-03-25T19:24:08.859 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:08.859 INFO:teuthology.orchestra.run.smithi142.stdout:Total 358 kB/s | 5.9 MB 00:17 2024-03-25T19:24:08.859 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:24:08.863 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:24:08.863 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:24:08.935 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:24:08.935 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:24:09.669 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:24:09.782 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:10.377 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:10.646 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:10.646 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:10.646 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:24:10.646 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:10.646 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:10.646 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:24:10.850 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:24:11.411 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:10:56 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:24:11.562 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:24:11.563 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:11.564 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 109 k 2024-03-25T19:24:11.564 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 440 k 2024-03-25T19:24:11.564 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:24:12.236 INFO:teuthology.orchestra.run.smithi142.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 162 kB/s | 109 kB 00:00 2024-03-25T19:24:12.236 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:12.236 INFO:teuthology.orchestra.run.smithi142.stdout:Total 162 kB/s | 109 kB 00:00 2024-03-25T19:24:12.236 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:24:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:24:12.240 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:24:12.275 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:24:12.275 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:24:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:24:12.492 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:12.871 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:13.135 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:13.135 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:13.135 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:24:13.136 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:13.136 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:13.136 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:24:13.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:24:13.893 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:10:58 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:24:14.042 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 5.4 M 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 25 M 2024-03-25T19:24:14.043 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:24:28.595 INFO:teuthology.orchestra.run.smithi142.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 5.4 MB 00:14 2024-03-25T19:24:28.596 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:28.596 INFO:teuthology.orchestra.run.smithi142.stdout:Total 377 kB/s | 5.4 MB 00:14 2024-03-25T19:24:28.597 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:24:28.600 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:24:28.601 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:24:28.669 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:24:28.669 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:24:29.332 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:24:29.452 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:29.962 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:30.242 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:30.243 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:30.243 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:24:30.243 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:30.243 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:30.243 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:24:30.454 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librados2-debuginfo 2024-03-25T19:24:31.022 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:11:16 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:24:31.170 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:24:31.171 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:31.171 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:24:31.171 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:31.171 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:24:31.171 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 44 M 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 253 M 2024-03-25T19:24:31.172 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:25:38.950 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 752 kB/s | 99 MB 02:14 2024-03-25T19:25:38.951 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:38.951 INFO:teuthology.orchestra.run.smithi047.stdout:Total 752 kB/s | 99 MB 02:14 2024-03-25T19:25:38.951 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:25:38.955 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:25:38.955 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:25:39.592 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:25:39.592 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:25:45.338 INFO:teuthology.orchestra.run.smithi142.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 611 kB/s | 44 MB 01:14 2024-03-25T19:25:45.338 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:45.339 INFO:teuthology.orchestra.run.smithi142.stdout:Total 611 kB/s | 44 MB 01:14 2024-03-25T19:25:45.339 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:25:45.342 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:25:45.343 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:25:45.641 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:25:45.641 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:25:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:25:50.600 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:50.694 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:25:50.803 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:25:52.583 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:25:52.911 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:25:52.912 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:25:52.912 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:25:52.912 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:52.912 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:25:52.912 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:25:53.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:25:53.565 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:53.695 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:12:38 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:25:53.834 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:53.834 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:25:53.835 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:25:53.835 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:53.835 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:25:53.835 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:25:53.842 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:25:53.843 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:25:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:25:53.843 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:25:53.843 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:25:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:25:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:25:53.844 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:25:53.844 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:25:53.844 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:25:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:25:53.844 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 52 M 2024-03-25T19:25:53.845 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 308 M 2024-03-25T19:25:53.845 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:25:54.015 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:25:54.586 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:12:41 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:25:54.740 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:25:54.740 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:25:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:25:54.741 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 287 M 2024-03-25T19:25:54.742 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.2 G 2024-03-25T19:25:54.742 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:26:43.284 INFO:teuthology.orchestra.run.smithi142.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 52 MB 00:49 2024-03-25T19:26:43.284 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:43.285 INFO:teuthology.orchestra.run.smithi142.stdout:Total 1.0 MB/s | 52 MB 00:49 2024-03-25T19:26:43.285 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:26:43.289 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:26:43.289 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:26:43.634 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:26:43.634 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:26:49.631 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:26:49.748 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:51.776 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:52.034 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:52.034 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:52.034 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:26:52.034 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:52.034 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:52.034 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:26:52.230 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:26:52.792 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:13:37 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:26:52.942 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:26:52.943 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:52.943 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:26:52.943 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:52.943 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:26:52.943 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 615 k 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 2.1 M 2024-03-25T19:26:52.944 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:26:54.394 INFO:teuthology.orchestra.run.smithi142.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 424 kB/s | 615 kB 00:01 2024-03-25T19:26:54.395 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:54.395 INFO:teuthology.orchestra.run.smithi142.stdout:Total 424 kB/s | 615 kB 00:01 2024-03-25T19:26:54.395 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:26:54.399 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:26:54.399 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:26:54.440 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:26:54.441 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:26:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:26:54.728 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:26:55.119 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:26:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:26:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:55.376 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:26:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:55.376 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:26:55.616 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:26:56.181 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:13:41 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:26:56.333 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:56.334 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:26:56.335 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:56.335 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:26:56.335 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:56.335 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 771 k 2024-03-25T19:26:56.335 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 2.7 M 2024-03-25T19:26:56.335 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:26:57.931 INFO:teuthology.orchestra.run.smithi142.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 483 kB/s | 771 kB 00:01 2024-03-25T19:26:57.931 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:57.932 INFO:teuthology.orchestra.run.smithi142.stdout:Total 483 kB/s | 771 kB 00:01 2024-03-25T19:26:57.932 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:26:57.936 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:26:57.936 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:26:57.978 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:26:57.978 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:26:58.165 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:26:58.263 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:26:58.630 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:26:58.893 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:26:58.893 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:58.893 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:26:58.893 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:58.893 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:58.893 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:26:59.122 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:26:59.688 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:13:44 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:26:59.838 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:26:59.838 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 1.1 M 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 4.3 M 2024-03-25T19:26:59.839 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:27:01.734 INFO:teuthology.orchestra.run.smithi142.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 608 kB/s | 1.1 MB 00:01 2024-03-25T19:27:01.734 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:01.734 INFO:teuthology.orchestra.run.smithi142.stdout:Total 607 kB/s | 1.1 MB 00:01 2024-03-25T19:27:01.735 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:27:01.739 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:27:01.739 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:27:01.782 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:27:01.782 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:27:02.001 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:27:02.107 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:02.767 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:02.768 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:02.768 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:27:02.768 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:02.768 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:02.768 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:27:03.006 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:27:03.571 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:13:48 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:27:03.720 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:27:03.721 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:03.722 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:27:03.722 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:03.722 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 337 k 2024-03-25T19:27:03.722 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 1.0 M 2024-03-25T19:27:03.722 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:27:04.895 INFO:teuthology.orchestra.run.smithi142.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 288 kB/s | 337 kB 00:01 2024-03-25T19:27:04.895 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:04.895 INFO:teuthology.orchestra.run.smithi142.stdout:Total 287 kB/s | 337 kB 00:01 2024-03-25T19:27:04.896 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:27:04.899 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:27:04.899 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:27:04.938 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:27:04.939 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:27:05.078 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:27:05.193 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:05.571 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:05.840 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:05.840 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:05.840 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:27:05.840 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:05.841 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:05.841 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:27:06.075 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:27:06.639 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:13:51 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:27:06.789 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:27:06.790 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:06.791 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:27:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:06.791 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 509 k 2024-03-25T19:27:06.791 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 2.5 M 2024-03-25T19:27:06.791 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:27:08.421 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 312 kB/s | 509 kB 00:01 2024-03-25T19:27:08.423 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:08.423 INFO:teuthology.orchestra.run.smithi142.stdout:Total 312 kB/s | 509 kB 00:01 2024-03-25T19:27:08.423 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:27:08.427 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:27:08.427 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:27:08.468 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:27:08.469 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:27:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:27:08.740 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:09.142 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:09.399 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:09.400 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:09.400 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:27:09.400 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:09.400 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:09.400 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:27:09.607 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:27:10.173 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:13:55 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:27:10.326 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 52 M 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 298 M 2024-03-25T19:27:10.327 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:28:12.435 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 850 kB/s | 52 MB 01:02 2024-03-25T19:28:12.435 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:12.436 INFO:teuthology.orchestra.run.smithi142.stdout:Total 850 kB/s | 52 MB 01:02 2024-03-25T19:28:12.436 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:28:12.439 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:28:12.439 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:28:12.775 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:28:12.775 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:28:18.697 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:28:18.949 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:28:20.979 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:28:21.246 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:28:21.246 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:28:21.246 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:28:21.246 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:21.246 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:28:21.247 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:28:21.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:28:22.024 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1:15:07 ago on Mon 25 Mar 2024 06:13:15 PM UTC. 2024-03-25T19:28:22.179 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:28:22.180 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-25T19:28:22.181 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-25T19:28:22.181 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2024-03-25T19:28:22.181 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:28:22.181 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 1.3 M 2024-03-25T19:28:22.181 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 5.5 M 2024-03-25T19:28:22.181 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-25T19:28:25.235 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 420 kB/s | 1.3 MB 00:03 2024-03-25T19:28:25.236 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:25.237 INFO:teuthology.orchestra.run.smithi142.stdout:Total 420 kB/s | 1.3 MB 00:03 2024-03-25T19:28:25.237 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-25T19:28:25.241 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-25T19:28:25.241 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-25T19:28:25.287 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-25T19:28:25.288 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-25T19:28:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-25T19:28:25.651 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:26.033 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:28:26.335 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-25T19:28:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:28:26.336 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-25T19:28:26.570 DEBUG:teuthology.parallel:result is None 2024-03-25T19:30:39.491 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 287 MB 04:44 2024-03-25T19:30:39.491 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:39.491 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.0 MB/s | 287 MB 04:44 2024-03-25T19:30:39.492 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:30:39.494 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:30:39.494 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:30:41.285 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:30:41.285 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:31:12.400 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:31:12.505 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:19.940 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:20.227 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:20.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:20.227 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:31:20.227 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:20.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:20.227 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:31:20.459 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:31:21.035 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:18:08 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:31:21.190 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:31:21.191 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:21.191 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:31:21.191 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:21.191 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:31:21.191 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 5.9 M 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 28 M 2024-03-25T19:31:21.192 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:31:24.789 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.7 MB/s | 5.9 MB 00:03 2024-03-25T19:31:24.790 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:24.791 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.7 MB/s | 5.9 MB 00:03 2024-03-25T19:31:24.791 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:31:24.795 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:31:24.795 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:31:24.870 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:31:24.870 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:31:25.652 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:31:25.768 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:26.341 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:26.607 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:31:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:26.608 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:31:26.837 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:31:27.412 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:18:14 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:31:27.570 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:31:27.570 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:27.570 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:31:27.570 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:27.571 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 109 k 2024-03-25T19:31:27.572 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 440 k 2024-03-25T19:31:27.572 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:31:28.376 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 135 kB/s | 109 kB 00:00 2024-03-25T19:31:28.377 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:28.377 INFO:teuthology.orchestra.run.smithi047.stdout:Total 135 kB/s | 109 kB 00:00 2024-03-25T19:31:28.377 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:31:28.381 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:31:28.381 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:31:28.419 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:31:28.419 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:31:28.545 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:31:28.652 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:29.014 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:29.323 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:29.323 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:29.323 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:31:29.323 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:29.323 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:29.323 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:31:29.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:31:30.114 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:18:17 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:31:30.269 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:31:30.270 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:30.271 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:31:30.271 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:30.271 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 5.4 M 2024-03-25T19:31:30.271 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 25 M 2024-03-25T19:31:30.271 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:31:38.320 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 681 kB/s | 5.4 MB 00:08 2024-03-25T19:31:38.320 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:38.321 INFO:teuthology.orchestra.run.smithi047.stdout:Total 681 kB/s | 5.4 MB 00:08 2024-03-25T19:31:38.321 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:31:38.325 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:31:38.325 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:31:38.398 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:31:38.398 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:31:39.097 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:31:39.264 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:31:39.762 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:31:40.030 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:31:40.031 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:40.031 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:31:40.031 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:40.031 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:40.031 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:31:40.233 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2-debuginfo 2024-03-25T19:31:40.805 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:18:27 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:31:40.962 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:31:40.962 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 44 M 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 253 M 2024-03-25T19:31:40.963 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:32:22.244 INFO:teuthology.orchestra.run.smithi047.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.1 MB/s | 44 MB 00:41 2024-03-25T19:32:22.245 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:22.245 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.1 MB/s | 44 MB 00:41 2024-03-25T19:32:22.245 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:32:22.248 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:32:22.249 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:32:22.562 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:32:22.562 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:32:28.021 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:32:28.120 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:29.904 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:32:30.175 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:32:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:30.175 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:32:30.175 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:32:30.384 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:32:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:19:17 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:32:31.109 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:32:31.110 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:32:31.111 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:32:31.111 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:32:31.111 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 52 M 2024-03-25T19:32:31.111 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 308 M 2024-03-25T19:32:31.111 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:32:53.614 INFO:teuthology.orchestra.run.smithi047.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 52 MB 00:22 2024-03-25T19:32:53.614 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:53.614 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.3 MB/s | 52 MB 00:22 2024-03-25T19:32:53.614 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:32:53.616 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:32:53.617 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:32:53.978 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:32:53.978 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:00.315 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:00.421 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:02.478 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:02.745 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:02.745 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:02.745 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:02.745 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:02.745 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:02.745 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:02.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:33:03.522 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:19:50 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:03.679 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:03.679 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:03.679 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 615 k 2024-03-25T19:33:03.679 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.1 M 2024-03-25T19:33:03.679 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:05.230 INFO:teuthology.orchestra.run.smithi047.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 396 kB/s | 615 kB 00:01 2024-03-25T19:33:05.231 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:05.232 INFO:teuthology.orchestra.run.smithi047.stdout:Total 396 kB/s | 615 kB 00:01 2024-03-25T19:33:05.232 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:05.236 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:05.236 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:05.279 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:05.280 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:05.582 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:06.360 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:06.360 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:06.360 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:06.360 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:06.360 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:06.360 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:06.578 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:33:07.151 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:19:54 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:07.310 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:07.310 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 771 k 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.7 M 2024-03-25T19:33:07.311 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:09.575 INFO:teuthology.orchestra.run.smithi047.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 341 kB/s | 771 kB 00:02 2024-03-25T19:33:09.575 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:09.576 INFO:teuthology.orchestra.run.smithi047.stdout:Total 340 kB/s | 771 kB 00:02 2024-03-25T19:33:09.576 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:09.580 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:09.580 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:09.625 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:09.625 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:09.819 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:09.919 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:10.553 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:10.553 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:10.553 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:10.553 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:10.554 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:10.554 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:10.800 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:33:11.371 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:19:58 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:11.527 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:11.528 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:11.529 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:11.529 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:11.529 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:11.529 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.1 M 2024-03-25T19:33:11.529 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 4.3 M 2024-03-25T19:33:11.529 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:12.536 INFO:teuthology.orchestra.run.smithi047.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 1.1 MB/s | 1.1 MB 00:01 2024-03-25T19:33:12.537 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:12.537 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.1 MB/s | 1.1 MB 00:01 2024-03-25T19:33:12.537 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:12.541 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:12.541 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:12.587 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:12.587 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:12.832 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:13.342 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:13.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:13.610 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:13.610 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:13.610 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:13.610 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:13.610 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:13.842 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:33:14.415 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:20:01 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:14.571 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:14.571 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:14.572 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 337 k 2024-03-25T19:33:14.573 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.0 M 2024-03-25T19:33:14.573 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:15.681 INFO:teuthology.orchestra.run.smithi047.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 304 kB/s | 337 kB 00:01 2024-03-25T19:33:15.682 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:15.683 INFO:teuthology.orchestra.run.smithi047.stdout:Total 304 kB/s | 337 kB 00:01 2024-03-25T19:33:15.683 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:15.686 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:15.687 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:15.729 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:15.729 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:15.891 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:16.009 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:16.405 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:16.659 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:16.660 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:16.660 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:16.660 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:16.660 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:16.660 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:16.896 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:33:17.469 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:20:04 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:17.626 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:17.627 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:17.628 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:17.628 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:17.628 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:17.628 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 509 k 2024-03-25T19:33:17.628 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.5 M 2024-03-25T19:33:17.628 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:18.368 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 688 kB/s | 509 kB 00:00 2024-03-25T19:33:18.368 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:18.369 INFO:teuthology.orchestra.run.smithi047.stdout:Total 687 kB/s | 509 kB 00:00 2024-03-25T19:33:18.369 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:18.372 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:18.372 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:18.413 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:18.413 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:18.578 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:18.695 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:19.100 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:19.377 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:19.377 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:19.377 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:19.377 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:19.377 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:19.377 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:19.593 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:33:20.164 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:20:07 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:20.319 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:20.320 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:20.321 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 52 M 2024-03-25T19:33:20.321 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 298 M 2024-03-25T19:33:20.321 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:34.969 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 52 MB 00:14 2024-03-25T19:33:34.970 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:34.970 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.5 MB/s | 52 MB 00:14 2024-03-25T19:33:34.971 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:34.975 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:34.975 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:35.327 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:35.327 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:41.588 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:41.696 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:33:43.723 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:33:43.990 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:33:43.990 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:43.990 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:43.990 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:43.991 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:43.991 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:44.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:33:44.790 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 1:20:31 ago on Mon 25 Mar 2024 06:13:13 PM UTC. 2024-03-25T19:33:44.945 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-25T19:33:44.946 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:44.947 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.3 M 2024-03-25T19:33:44.947 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 5.5 M 2024-03-25T19:33:44.947 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-25T19:33:47.255 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 556 kB/s | 1.3 MB 00:02 2024-03-25T19:33:47.255 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:47.256 INFO:teuthology.orchestra.run.smithi047.stdout:Total 555 kB/s | 1.3 MB 00:02 2024-03-25T19:33:47.256 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-25T19:33:47.260 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-25T19:33:47.260 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-25T19:33:47.308 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-25T19:33:47.308 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-25T19:33:47.560 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-25T19:33:47.673 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:48.093 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:48.365 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:48.365 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:48.365 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-25T19:33:48.365 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:48.365 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:48.365 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-25T19:33:48.590 DEBUG:teuthology.parallel:result is None 2024-03-25T19:33:48.591 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:33:48.811 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:33:48.831 INFO:teuthology.orchestra.run.smithi047.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:33:48.832 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:33:48.832 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:33:48.834 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:33:48.977 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:33:49.001 INFO:teuthology.orchestra.run.smithi142.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:33:49.001 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:33:49.001 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:33:49.003 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:33:49.003 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:49.003 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:33:49.032 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:49.032 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:33:49.072 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:33:49.072 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:49.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:33:49.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:33:49.167 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:49.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:33:49.197 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:33:49.267 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:33:49.267 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:49.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:33:49.294 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:33:49.362 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:49.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:33:49.393 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:33:49.463 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:33:49.463 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:49.463 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:33:49.490 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:33:49.558 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:49.558 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:33:49.587 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:33:49.656 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:33:49.739 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:33:49.739 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2024-03-25T19:33:49.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /var/log/ceph 2024-03-25T19:33:49.769 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:33:49.769 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:33:49.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:33:49.836 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:33:49.836 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:33:49.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:33:49.915 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:33:49.916 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:33:49.916 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621320', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFN2YZrooIvVsISkt9MTFOIWr/dG0HSHOvYeYXmO+8d07avH0ttJ1lX1LOopxikblA5JwqPr0bIC6G6CEjmTZ5I=', 'smithi142.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFignQ13rXWoi83NQO6kOItIIJTX6k/evnxiJbbCD4HIK+LWHM0J7ITDwhp08F5/ptql0cVFiEQLX4gfn4ga9j4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:33:49.916 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:33:49.945 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:33:49.960 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:33:50.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:33:50.027 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:50.027 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:33:50.082 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:33:50.082 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-03-25T19:33:50.137 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:33:50.137 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:50.137 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T19:33:50.137 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:50.137 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:50.137 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-25 18:18:35.336775445 +0000 2024-03-25T19:33:50.138 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-25 18:11:27.062892664 +0000 2024-03-25T19:33:50.138 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-25 18:11:27.062892664 +0000 2024-03-25T19:33:50.138 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-25 18:11:27.062892664 +0000 2024-03-25T19:33:50.138 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-25T19:33:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.5591e-05 s, 5.4 MB/s 2024-03-25T19:33:50.205 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:33:50.261 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-03-25T19:33:50.316 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-25 18:18:35.336775445 +0000 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-25 18:11:27.400885267 +0000 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-25 18:11:27.400885267 +0000 2024-03-25T19:33:50.317 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-25 18:11:27.400885267 +0000 2024-03-25T19:33:50.318 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:33:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-25T19:33:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-25T19:33:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.00010081 s, 5.1 MB/s 2024-03-25T19:33:50.384 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:33:50.442 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-25 18:18:35.337775423 +0000 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-25 18:11:27.736877916 +0000 2024-03-25T19:33:50.497 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-25 18:11:27.736877916 +0000 2024-03-25T19:33:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-25 18:11:27.736877916 +0000 2024-03-25T19:33:50.498 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-25T19:33:50.563 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.6443e-05 s, 5.3 MB/s 2024-03-25T19:33:50.565 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:33:50.621 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-25 18:18:35.337775423 +0000 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-25 18:11:28.059870848 +0000 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-25 18:11:28.059870848 +0000 2024-03-25T19:33:50.676 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-25 18:11:28.059870848 +0000 2024-03-25T19:33:50.676 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:33:50.741 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-25T19:33:50.741 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-25T19:33:50.742 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.5447e-05 s, 5.4 MB/s 2024-03-25T19:33:50.743 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:33:50.799 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:50.799 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:50.799 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:33:50.815 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:33:50.815 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-03-25T19:33:50.871 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:33:50.871 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:50.871 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:33:50.871 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:50.872 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:50.872 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-25 18:18:54.848069857 +0000 2024-03-25T19:33:50.872 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-25 18:11:13.487782588 +0000 2024-03-25T19:33:50.872 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-25 18:11:13.487782588 +0000 2024-03-25T19:33:50.872 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-25 18:11:13.487782588 +0000 2024-03-25T19:33:50.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:33:50.938 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-25T19:33:50.938 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-25T19:33:50.938 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000214094 s, 2.4 MB/s 2024-03-25T19:33:50.939 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:33:50.995 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-03-25T19:33:51.050 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:33:51.050 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-25 18:18:54.849069838 +0000 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-25 18:11:13.822776149 +0000 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-25 18:11:13.822776149 +0000 2024-03-25T19:33:51.051 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-25 18:11:13.822776149 +0000 2024-03-25T19:33:51.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:33:51.116 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-25T19:33:51.116 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-25T19:33:51.116 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000249208 s, 2.1 MB/s 2024-03-25T19:33:51.117 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:33:51.173 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-03-25T19:33:51.227 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:33:51.227 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:51.227 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:33:51.228 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:51.228 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:51.228 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-25 18:18:54.849069838 +0000 2024-03-25T19:33:51.228 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-25 18:11:14.158769691 +0000 2024-03-25T19:33:51.228 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-25 18:11:14.158769691 +0000 2024-03-25T19:33:51.228 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-25 18:11:14.158769691 +0000 2024-03-25T19:33:51.228 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:33:51.292 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-25T19:33:51.293 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-25T19:33:51.293 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000155378 s, 3.3 MB/s 2024-03-25T19:33:51.294 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:33:51.349 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-25 18:18:54.850069819 +0000 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-25 18:11:14.484763424 +0000 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-25 18:11:14.484763424 +0000 2024-03-25T19:33:51.404 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-25 18:11:14.484763424 +0000 2024-03-25T19:33:51.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:33:51.469 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-25T19:33:51.469 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-25T19:33:51.469 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000211154 s, 2.4 MB/s 2024-03-25T19:33:51.470 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:33:51.525 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:51.526 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:33:51.526 INFO:tasks.ceph:Generating config... 2024-03-25T19:33:51.527 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:33:51.527 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:33:51.527 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:33:51.527 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:33:51.527 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:33:51.527 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:33:51.527 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:33:51.527 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:33:51.527 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:33:51.527 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:33:51.528 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:33:51.528 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:33:51.528 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:33:51.528 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:33:51.529 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:33:51.530 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:33:51.531 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:33:51.531 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:33:51.589 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:33:51.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:33:51.639 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:33:51.666 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.47:6789]'), ('mon.c', '[v1:172.21.15.47:6790]'), ('mon.b', '[v1:172.21.15.142:6789]')] 2024-03-25T19:33:51.667 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.142:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:33:51.668 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:51.668 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:33:51.721 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.47:6789]' --addv c '[v1:172.21.15.47:6790]' --addv b '[v1:172.21.15.142:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:51.806 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: generated fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = quincy 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 0 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:min_mon_release 17 (quincy) 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:election_strategy: 3 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:33:51.807 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:33:51.808 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:33:51.822 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ff41e6e4-140a-4dba-9564-ead70286575a... 2024-03-25T19:33:51.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:33:51.866 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:33:51.913 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-03-25T19:33:51.913 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-03-25T19:33:51.913 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:33:51.913 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-03-25T19:33:51.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-03-25T19:33:51.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-03-25T19:33:51.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.142:6789]" 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon client directed command retry = 5 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: mon election default strategy = 3 2024-03-25T19:33:51.916 INFO:teuthology.orchestra.run.smithi047.stdout: ms bind msgr2 = False 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject socket failures = 5000 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: ms type = async 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd heartbeat grace = 80 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-03-25T19:33:51.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-03-25T19:33:51.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = stupid 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore zero block detection = True 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1 2024-03-25T19:33:51.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 10 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: osd class default list = * 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: osd class load list = * 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: osd client watch timeout = 120 2024-03-25T19:33:51.920 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify cached snaps = True 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify missing on start = True 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd fast shutdown = False 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd max markdown count = 1000 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock override recovery settings = True 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue = debug_random 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue cut off = debug_random 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: osd shutdown pgref assert = True 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-03-25T19:33:51.921 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.922 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon min osdmap epochs = 50 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd crush smoke test = False 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune interval = 2 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune min = 15 2024-03-25T19:33:51.923 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: mon scrub interval = 300 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on pool no app = False 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: paxos service trim min = 10 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: debug objecter = 20 2024-03-25T19:33:51.924 INFO:teuthology.orchestra.run.smithi047.stdout: debug rados = 20 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi142.stdout:[global] 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi142.stdout: chdir = "" 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi142.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi142.stdout: auth supported = cephx 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.925 INFO:teuthology.orchestra.run.smithi142.stdout: filestore xattr use omap = true 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: mon clock drift allowed = 1.000 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: osd crush chooseleaf type = 0 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: auth debug = true 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on old message = true 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on bug = true 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: mon pg warn max object skew = 0 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.926 INFO:teuthology.orchestra.run.smithi142.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default size = 2 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow primary affinity = true 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow pg remap = true 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:33:51.927 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on no sortbitwise = false 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on too few osds = false 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon_allow_pool_size_one = true 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: osd default data pool replay window = 5 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon allow pool delete = true 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: mon cluster log file level = debug 2024-03-25T19:33:51.928 INFO:teuthology.orchestra.run.smithi142.stdout: debug asserts on shutdown = true 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: mon health detail to clog = false 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: mon host = "[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.142:6789]" 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: mon client directed command retry = 5 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: mon election default strategy = 3 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: ms bind msgr2 = False 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject socket failures = 5000 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: ms type = async 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: osd heartbeat grace = 80 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: fsid = ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout:[osd] 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: osd journal size = 100 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.929 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub load threshold = 5.0 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub max interval = 600 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd recover clone overlap = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd recovery max chunk = 1048576 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug shutdown = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug op order = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify stray on activate = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug trim objects = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd open classes on start = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug pg log writeout = true 2024-03-25T19:33:51.930 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: osd map max advance = 10 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: journal zero on create = true 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: filestore apply finisher threads = 3 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: bdev debug aio = true 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug misdirected ops = true 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: bdev async discard = True 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: bdev enable discard = True 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore allocator = stupid 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore block size = 96636764160 2024-03-25T19:33:51.931 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore fsck on mount = True 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore zero block detection = True 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluefs = 1 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluestore = 1 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: debug monc = 20 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: debug osd = 20 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: debug rocksdb = 10 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd full ratio = 0.9 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: osd class default list = * 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: osd class load list = * 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: osd client watch timeout = 120 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify cached snaps = True 2024-03-25T19:33:51.932 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify missing on start = True 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd fast shutdown = False 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd max markdown count = 1000 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd mclock override recovery settings = True 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd objectstore = bluestore 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd op queue = debug_random 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd op queue cut off = debug_random 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: osd shutdown pgref assert = True 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr] 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: debug mgr = 20 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2024-03-25T19:33:51.933 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mgr/telemetry/nag = false 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout:[mon] 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: debug paxos = 20 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon data avail warn = 5 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon mgr mkfs grace = 240 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd reporter subtree level = osd 2024-03-25T19:33:51.934 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd prime pg temp = true 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: # 1m isn't quite enough 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon min osdmap epochs = 50 2024-03-25T19:33:51.935 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd crush smoke test = False 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: mon osdmap full prune interval = 2 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: mon osdmap full prune min = 15 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: mon scrub interval = 300 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on pool no app = False 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: paxos service trim min = 10 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout:[client] 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: rgw cache enabled = true 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable ops log = true 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable usage log = true 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:33:51.936 INFO:teuthology.orchestra.run.smithi142.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout: debug monc = 20 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout: debug objecter = 20 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout: debug rados = 20 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.a] 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.c] 2024-03-25T19:33:51.937 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.b] 2024-03-25T19:33:51.938 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:33:51.939 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:33:51.993 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:33:51.993 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:51.993 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:33:52.007 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:52.007 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:33:52.061 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:33:52.061 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:52.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:33:52.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:33:52.140 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:33:52.140 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:52.194 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:33:52.195 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:52.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:33:52.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:33:52.234 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:33:52.234 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:52.289 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:33:52.289 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:33:52.290 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:33:52.344 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:33:52.346 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:33:52.408 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:33:52.410 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:33:52.410 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:33:52.411 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:33:52.452 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:33:52.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:33:52.510 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:33:52.525 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:33:52.525 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:33:52.525 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:33:52.551 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:52.552 INFO:tasks.ceph:role: osd.0 2024-03-25T19:33:52.552 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:33:52.552 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:52.618 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:52.651 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-25T19:33:52.653 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-25T19:33:52.653 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:33:52.728 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:33:52.797 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:52.864 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:52.864 INFO:tasks.ceph:role: osd.1 2024-03-25T19:33:52.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:33:52.864 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:33:52.930 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:52.930 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:52.930 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:52.930 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:52.931 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:52.963 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-25T19:33:52.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-25T19:33:52.965 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:53.036 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:53.107 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:53.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:53.174 INFO:tasks.ceph:role: osd.2 2024-03-25T19:33:53.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:33:53.174 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:53.240 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:53.272 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-25T19:33:53.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-25T19:33:53.274 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:53.350 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:33:53.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:53.489 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:33:53.490 INFO:tasks.ceph:role: osd.3 2024-03-25T19:33:53.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:33:53.490 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:33:53.557 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:33:53.589 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-25T19:33:53.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-25T19:33:53.591 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:53.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:33:53.733 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:53.826+0000 7f49a4a2a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:33:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:53.826+0000 7f49a4a2a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:33:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:53.826+0000 7f49a4a2a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:33:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:53.826+0000 7f49a4a2a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:33:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:53.826+0000 7f49a4a2a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:33:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:53.827+0000 7f49a4a2a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:33:56.670 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:33:56.701 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:56.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:56.793+0000 7fa234213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:33:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:56.793+0000 7fa234213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:33:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:56.793+0000 7fa234213640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:33:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:56.793+0000 7fa234213640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:33:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:56.793+0000 7fa234213640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:33:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:56.793+0000 7fa234213640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:33:59.630 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:33:59.661 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:33:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:59.751+0000 7f5b2bc37640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:33:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:59.751+0000 7f5b2bc37640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:33:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:59.751+0000 7f5b2bc37640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:33:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:59.751+0000 7f5b2bc37640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:33:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:59.751+0000 7f5b2bc37640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:33:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:33:59.752+0000 7f5b2bc37640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:34:02.589 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:34:02.619 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:02.711+0000 7fa9bb660640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:34:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:02.711+0000 7fa9bb660640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:34:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:02.711+0000 7fa9bb660640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:34:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:02.711+0000 7fa9bb660640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:34:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:02.711+0000 7fa9bb660640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:34:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:02.711+0000 7fa9bb660640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:34:05.546 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:34:05.576 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:05.608 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:05.608 INFO:tasks.ceph:role: osd.4 2024-03-25T19:34:05.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:34:05.608 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:34:05.685 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:05.685 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:05.686 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:05.686 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:05.686 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:05.686 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:05.686 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:06.194 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2024-03-25T19:34:06.196 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-03-25T19:34:06.196 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:06.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:06.391 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:06.461 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:06.462 INFO:tasks.ceph:role: osd.5 2024-03-25T19:34:06.462 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:34:06.462 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:06.538 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:06.988 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2024-03-25T19:34:06.990 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-03-25T19:34:06.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:07.103 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:07.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:07.208 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:07.208 INFO:tasks.ceph:role: osd.6 2024-03-25T19:34:07.208 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:34:07.208 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:34:07.285 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:07.285 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:07.285 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:07.285 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:07.285 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:07.285 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:07.286 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:07.286 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:07.286 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:07.286 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:07.771 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2024-03-25T19:34:07.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-03-25T19:34:07.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:07.916 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:07.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:08.022 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:08.023 INFO:tasks.ceph:role: osd.7 2024-03-25T19:34:08.023 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:34:08.023 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:34:08.100 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:08.101 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:08.629 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2024-03-25T19:34:08.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-03-25T19:34:08.631 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:08.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:08.790 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:08.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:08.895+0000 7f9583013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:34:08.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:08.895+0000 7f9583013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:34:08.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:08.895+0000 7f9583013640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:34:08.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:08.895+0000 7f9583013640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:34:08.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:08.895+0000 7f9583013640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:34:08.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:08.896+0000 7f9583013640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:34:11.734 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:11.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:11.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:11.856+0000 7fec7e80f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:34:11.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:11.856+0000 7fec7e80f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:34:11.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:11.856+0000 7fec7e80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:34:11.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:11.856+0000 7fec7e80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:34:11.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:11.856+0000 7fec7e80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:34:11.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:11.857+0000 7fec7e80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:34:14.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:14.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:14.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:14.820+0000 7f8672a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:34:14.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:14.821+0000 7f8672a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:34:14.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:14.821+0000 7f8672a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:34:14.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:14.821+0000 7f8672a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:34:14.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:14.821+0000 7f8672a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:34:14.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:14.821+0000 7f8672a11640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:34:17.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:17.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:17.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:17.784+0000 7fc9fedff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:34:17.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:17.784+0000 7fc9fedff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:34:17.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:17.784+0000 7fc9fedff640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:34:17.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:17.784+0000 7fc9fedff640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:34:17.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:17.784+0000 7fc9fedff640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:34:17.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:17.785+0000 7fc9fedff640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:34:20.631 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:20.662 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:34:20.662 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:20.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:34:20.694 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:20.694 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:34:20.765 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:20.765 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:34:20.835 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:20.835 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:34:20.904 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:20.904 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:34:20.973 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:20.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:34:21.003 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:21.003 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:34:21.073 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:21.073 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:34:21.142 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:21.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:34:21.211 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:21.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:34:21.278 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:21.278 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:34:21.294 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:21.294 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:34:21.333 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:34:21.333 DEBUG:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:34:21.336 DEBUG:teuthology.orchestra.run.smithi142:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.y] 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCg0QFmXwCXFBAAK8mqwe3bXWTWJ1lpowMcdg== 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCh0QFmH+5RMRAAiXOuBg9eBjX4DK+LOtNvtA== 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCk0QFmZcdTLxAAyb50rv461yb4xAqP+QNjPg== 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCn0QFmqNnWLBAAcCE545V4BkSU6NmGBF+Vcg== 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.3] 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCq0QFmv3puKhAA41d03oe1wWsh35tBzpvp6w== 2024-03-25T19:34:21.376 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCg0QFm6bFqGBAArESAbHlSrTB7nKAdTPIZ1w== 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCw0QFmNk9wNRAA+Ok21iswDznU944bh9t7aw== 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCz0QFmkawgMxAAWPXWFdC3SM3A+U7TcVdDnw== 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.6] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQC20QFmM0n/MBAAXTIEYxlO1Jmfh8aqsIFrKQ== 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.7] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQC50QFmqgTWLhAAkP1bLw2dMV+ieDEx6ppW7w== 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCg0QFm+k0SGxAAk5FT6pLplOqieAVsshb3dA== 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout:[client.1] 2024-03-25T19:34:21.377 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQCg0QFmuS2AHhAA58FpgTzwxBg12fmLourUNw== 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.y] 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCg0QFmXwCXFBAAK8mqwe3bXWTWJ1lpowMcdg== 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.0] 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCh0QFmH+5RMRAAiXOuBg9eBjX4DK+LOtNvtA== 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.1] 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCk0QFmZcdTLxAAyb50rv461yb4xAqP+QNjPg== 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.2] 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCn0QFmqNnWLBAAcCE545V4BkSU6NmGBF+Vcg== 2024-03-25T19:34:21.398 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.3] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCq0QFmv3puKhAA41d03oe1wWsh35tBzpvp6w== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.x] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCg0QFm6bFqGBAArESAbHlSrTB7nKAdTPIZ1w== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.4] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCw0QFmNk9wNRAA+Ok21iswDznU944bh9t7aw== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.5] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCz0QFmkawgMxAAWPXWFdC3SM3A+U7TcVdDnw== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.6] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQC20QFmM0n/MBAAXTIEYxlO1Jmfh8aqsIFrKQ== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.7] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQC50QFmqgTWLhAAkP1bLw2dMV+ieDEx6ppW7w== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCg0QFm+k0SGxAAk5FT6pLplOqieAVsshb3dA== 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout:[client.1] 2024-03-25T19:34:21.399 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCg0QFmuS2AHhAA58FpgTzwxBg12fmLourUNw== 2024-03-25T19:34:21.400 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:34:21.420 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:34:21.492 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.546 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.599 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.601 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.653 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.707 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:34:21.709 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:34:21.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.814 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.816 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.868 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.922 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:34:21.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:34:21.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:34:22.031 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:34:22.034 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:34:22.088 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:34:22.088 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:34:22.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:34:22.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:34:22.236 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:34:22.304 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:34:22.400 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:34:22.426 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:34:22.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:34:22.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:34:22.578 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:22.580 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:22.633 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:34:22.633 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:34:22.633 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:34:22.633 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:34:22.636 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:34:22.636 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:34:22.636 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:34:22.636 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:34:22.638 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:34:22.638 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:34:22.638 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:34:22.638 DEBUG:teuthology.orchestra.run.smithi142:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:34:22.675 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:34:22.676 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:34:22.676 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:34:22.676 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:34:22.679 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:34:22.679 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:34:22.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:34:22.681 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:34:22.681 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:34:22.681 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.782+0000 7f491dc1e640 10 monclient: get_monmap_and_config 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.782+0000 7f491dc1e640 10 monclient: build_initial_monmap 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.782+0000 7f491dc1e640 10 monclient: monmap: 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:34:22.783850+0000 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:34:22.783850+0000 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:22.782 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:22.783 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:22.783 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:22.783 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:22.783 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.782+0000 7f491dc1e640 1 Processor -- start 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.782+0000 7f491dc1e640 1 -- start start 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.782+0000 7f491dc1e640 10 monclient: init 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:22.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient(hunting): picked mon.noname-b con 0x7f49180ad590 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient(hunting): picked mon.noname-a con 0x7f49180aaa10 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aee40 con 0x7f49180aaa10 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180ae980 con 0x7f49180ad1b0 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aeb10 con 0x7f49180ad590 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f491dc1e640 10 monclient(hunting): authenticate will time out at 5578.070446s 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f49180717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f4916ffd640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f49180ad590 legacy=0x7f49181401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:22.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.783+0000 7f49177fe640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4918072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:22.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.984+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f49180717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:22.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.984+0000 7f4916ffd640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f49180ad590 legacy=0x7f49181401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:22.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:22.984+0000 7f49177fe640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4918072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:23.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:23.385+0000 7f4916ffd640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f49180ad590 legacy=0x7f49181401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:23.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:23.385+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f49180717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:23.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:23.385+0000 7f49177fe640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4918072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:24.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:24.186+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f49180717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:24.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:24.186+0000 7f4916ffd640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f49180ad590 legacy=0x7f49181401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:24.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:24.186+0000 7f49177fe640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4918072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:25.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): tick 2024-03-25T19:34:25.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:34:25.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:34:25.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:34:25.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f49180ad590 legacy=0x7f49181401f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4918072540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f49180717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f4904003680 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f49180aaa10 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491814fb10 con 0x7f49180aaa10 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aee40 con 0x7f49180ad1b0 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180ae980 con 0x7f4904003680 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49157fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f490400cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:25.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.783+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f49040092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.984+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f490400cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.984+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:25.984+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f49040092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:26.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:26.385+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f490400cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:26.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:26.385+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:26.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:26.385+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f49040092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:27.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:27.186+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f490400cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:27.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:27.186+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:27.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:27.186+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f49040092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:28.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): tick 2024-03-25T19:34:28.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f49040092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f490400cab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f49180aaa10 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f4904003680 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aeb10 con 0x7f49180aaa10 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491814fb10 con 0x7f49180ad1b0 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aee40 con 0x7f4904003680 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49157fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:28.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:28.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.783+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4904009230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.984+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.984+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:28.984+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4904009230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:29.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:29.385+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:29.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:29.385+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:29.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:29.385+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4904009230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:29.654 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:00:06.844 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:29.959 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:00:07.149 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:30.138 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:00:07.374 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:30.139 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:00:07.375 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:30.186+0000 7f49177fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-25T19:34:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:30.186+0000 7f4917fff640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-03-25T19:34:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:30.186+0000 7f4916ffd640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4904009230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-03-25T19:34:30.465 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:00:07.701 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:30.467 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:00:07.702 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:31.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 10 monclient(hunting): tick 2024-03-25T19:34:31.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:34:31.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:34:31.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4904009230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.783+0000 7f49157fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f4904003680 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f49180aaa10 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180ae980 con 0x7f49180aaa10 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aeb10 con 0x7f49180ad1b0 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491814fb10 con 0x7f4904003680 2024-03-25T19:34:31.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.784+0000 7f49157fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:31.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.786+0000 7f49177fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 0x7f490400caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:40804/0 (socket says 172.21.15.47:40804) 2024-03-25T19:34:31.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:31.786+0000 7f49177fe640 1 -- 172.21.15.47:0/3679368897 learned_addr learned my addr 172.21.15.47:0/3679368897 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:32.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.174+0000 7f4916ffd640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:34:32.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.174+0000 7f4916ffd640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:34:32.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.174+0000 7f4916ffd640 1 --1- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 0x7f4904005ab0 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T19:34:32.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.174+0000 7f4916ffd640 1 --1- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 0x7f4904005ab0 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:34:32.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.174+0000 7f49167fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-03-25T19:34:32.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.176+0000 7f4917fff640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:34:32.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.176+0000 7f4917fff640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:34:32.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:32.176+0000 7f4917fff640 1 --1- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 0x7f4904010ad0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): tick 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f490400caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f49180aaa10 legacy=0x7f4904010ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:34.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f4904003680 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f4904010340 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180aee40 con 0x7f4904010340 2024-03-25T19:34:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4904010810 con 0x7f49180ad1b0 2024-03-25T19:34:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4904010b20 con 0x7f4904003680 2024-03-25T19:34:34.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:34.785+0000 7f49157fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): tick 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f4904003680 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f490400caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f4904010340 legacy=0x7f4904009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f490400d1e0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f4904010340 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4904010d50 con 0x7f4904010340 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490400dcd0 con 0x7f49180ad1b0 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490400de60 con 0x7f490400d1e0 2024-03-25T19:34:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:37.785+0000 7f49157fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:40.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): tick 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f490400d1e0 legacy=0x7f490400d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f4904005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f4904010340 legacy=0x7f490400caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f49040093d0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f4904008940 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4904009240 con 0x7f4904008940 2024-03-25T19:34:40.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4904008ec0 con 0x7f49180ad1b0 2024-03-25T19:34:40.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490400caa0 con 0x7f49040093d0 2024-03-25T19:34:40.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:40.785+0000 7f49157fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:41.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.012+0000 7f4917fff640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f490400d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:34:41.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.012+0000 7f4917fff640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f490400d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:34:41.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.012+0000 7f4917fff640 1 --1- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 0x7f490400d5c0 :-1 s=OPENED pgs=23 cs=1 l=1).handle_message read tag failed 2024-03-25T19:34:41.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.012+0000 7f4917fff640 1 --1- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 0x7f490400d5c0 :-1 s=OPENED pgs=23 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:34:41.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.012+0000 7f49167fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.142:6789/0 2024-03-25T19:34:41.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.237+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188406946 0 0) 0x7f4904009240 con 0x7f4904008940 2024-03-25T19:34:41.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.237+0000 7f49167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:41.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.237+0000 7f49167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:41.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.237+0000 7f49167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:41.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.237+0000 7f49167fc640 10 monclient(hunting): my global_id is 4098 2024-03-25T19:34:41.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.237+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4900003520 con 0x7f4904008940 2024-03-25T19:34:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.278+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707850985 0 0) 0x7f490400caa0 con 0x7f49040093d0 2024-03-25T19:34:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.278+0000 7f49167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.278+0000 7f49167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.278+0000 7f49167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.278+0000 7f49167fc640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:34:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.278+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904009240 con 0x7f49040093d0 2024-03-25T19:34:41.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198004422 0 0) 0x7f4900003520 con 0x7f4904008940 2024-03-25T19:34:41.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6790/0 conn(0x7f49040093d0 legacy=0x7f4904008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f490400d5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:34:41.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 1 monclient: found mon.noname-a 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490400cc30 con 0x7f4904008940 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.293+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f490c003f60 con 0x7f4904008940 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.294+0000 7f49167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.294+0000 7f49167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.294+0000 7f49167fc640 10 monclient: dump: 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:41.294 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.294+0000 7f491dc1e640 5 monclient: authenticate success, global_id 4098 2024-03-25T19:34:41.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.294+0000 7f491dc1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:41.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f490c0040c0 con 0x7f4904008940 2024-03-25T19:34:41.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f49167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:41.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f49167fc640 1 -- 172.21.15.47:0/3679368897 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f490c004510 con 0x7f4904008940 2024-03-25T19:34:41.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 4 set_mon_vals no callback set 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f49167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f49167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f49167fc640 10 monclient: dump: 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:41.382 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 10 monclient: shutdown 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 1 -- 172.21.15.47:0/3679368897 >> v1:172.21.15.47:6789/0 conn(0x7f4904008940 legacy=0x7f4904016470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 1 -- 172.21.15.47:0/3679368897 shutdown_connections 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 1 -- 172.21.15.47:0/3679368897 >> 172.21.15.47:0/3679368897 conn(0x7f491806b560 msgr2=0x7f49180b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 1 -- 172.21.15.47:0/3679368897 shutdown_connections 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.382+0000 7f491dc1e640 1 -- 172.21.15.47:0/3679368897 wait complete. 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.383+0000 7f491dc1e640 10 monclient: build_initial_monmap 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.383+0000 7f491dc1e640 10 monclient: monmap: 2024-03-25T19:34:41.385 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.383+0000 7f491dc1e640 1 librados: starting msgr at 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.383+0000 7f491dc1e640 1 librados: starting objecter 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.383+0000 7f491dc1e640 1 Processor -- start 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 1 -- start start 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 1 librados: setting wanted keys 2024-03-25T19:34:41.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 1 librados: calling monclient init 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient: init 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient(hunting): picked mon.noname-c con 0x7f4904008940 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient(hunting): picked mon.noname-a con 0x7f49040093d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient(hunting): picked mon.noname-b con 0x7f49180ad1b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918144180 con 0x7f49040093d0 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918070f30 con 0x7f49180ad1b0 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918142a70 con 0x7f4904008940 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:41.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.384+0000 7f491dc1e640 10 monclient(hunting): authenticate will time out at 5596.670802s 2024-03-25T19:34:41.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.390+0000 7f4916ffd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f49040093d0 0x7f49181dadf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60228/0 (socket says 172.21.15.47:60228) 2024-03-25T19:34:41.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.390+0000 7f4916ffd640 1 -- 172.21.15.47:0/3442055015 learned_addr learned my addr 172.21.15.47:0/3442055015 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:41.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.390+0000 7f48fffff640 10 client.?.objecter ms_handle_connect 0x7f49040093d0 2024-03-25T19:34:41.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.390+0000 7f48fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:41.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.396+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645859151 0 0) 0x7f4918144180 con 0x7f49040093d0 2024-03-25T19:34:41.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.396+0000 7f48fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:41.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.396+0000 7f48fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:41.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.396+0000 7f48fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:41.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.396+0000 7f48fffff640 10 monclient(hunting): my global_id is 4107 2024-03-25T19:34:41.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.396+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490400cdc0 con 0x7f49040093d0 2024-03-25T19:34:41.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.408+0000 7f48fffff640 10 client.?.objecter ms_handle_connect 0x7f49180ad1b0 2024-03-25T19:34:41.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.408+0000 7f48fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:41.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.413+0000 7f48fffff640 10 client.?.objecter ms_handle_connect 0x7f4904008940 2024-03-25T19:34:41.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.413+0000 7f48fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:41.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.428+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233319687 0 0) 0x7f4918070f30 con 0x7f49180ad1b0 2024-03-25T19:34:41.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.428+0000 7f48fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:41.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.428+0000 7f48fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:41.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.428+0000 7f48fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:41.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.428+0000 7f48fffff640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:34:41.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.428+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4918144180 con 0x7f49180ad1b0 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 148741245 0 0) 0x7f490400cdc0 con 0x7f49040093d0 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4918070f30 con 0x7f49040093d0 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f490c003f40 con 0x7f49040093d0 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 10 monclient(hunting): dump: 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:41.440 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:41.441 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:41.441 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:41.441 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:41.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.440+0000 7f48fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.452+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3599519563 0 0) 0x7f4918144180 con 0x7f49180ad1b0 2024-03-25T19:34:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.453+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f490400cdc0 con 0x7f49180ad1b0 2024-03-25T19:34:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.453+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4910004480 con 0x7f49180ad1b0 2024-03-25T19:34:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.453+0000 7f48fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.453+0000 7f48fffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.453+0000 7f48fffff640 10 monclient(hunting): dump: 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:41.453 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:41.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1101914576 0 0) 0x7f490400cdc0 con 0x7f49180ad1b0 2024-03-25T19:34:41.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 >> v1:172.21.15.47:6790/0 conn(0x7f4904008940 legacy=0x7f4918133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 >> v1:172.21.15.47:6789/0 conn(0x7f49040093d0 legacy=0x7f49181dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 1 monclient: found mon.b 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49181445f0 con 0x7f49180ad1b0 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 10 monclient: _finish_auth 0 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f48fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f491dc1e640 5 monclient: authenticate success, global_id 4111 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f491dc1e640 10 monclient: _renew_subs 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f491dc1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.463+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4918142d50 con 0x7f49180ad1b0 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:41.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 10 monclient: _renew_subs 2024-03-25T19:34:41.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:41.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f491806e070 con 0x7f49180ad1b0 2024-03-25T19:34:41.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 1 librados: init done 2024-03-25T19:34:41.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:41.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:41.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:41.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.464+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49180af960 con 0x7f49180ad1b0 2024-03-25T19:34:41.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4910004890 con 0x7f49180ad1b0 2024-03-25T19:34:41.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:41.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4910004ca0 con 0x7f49180ad1b0 2024-03-25T19:34:41.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:41.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 10 monclient: dump: 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4910005250 con 0x7f49180ad1b0 2024-03-25T19:34:41.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.714+0000 7f4915ffb640 4 set_mon_vals no callback set 2024-03-25T19:34:41.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 929948699 0 0) 0x7f4910006410 con 0x7f49180ad1b0 2024-03-25T19:34:41.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 10 client.4111.objecter ms_dispatch 0x7f4918000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:34:41.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 3 client.4111.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:34:41.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 3 client.4111.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:34:41.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-25T19:34:41.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4910006aa0 con 0x7f49180ad1b0 2024-03-25T19:34:41.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:41.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.717+0000 7f48fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.800+0000 7f491dc1e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:34:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.800+0000 7f491dc1e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:34:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.800+0000 7f491dc1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:41.800+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4918144310 con 0x7f49180ad1b0 2024-03-25T19:34:41.939 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:41.939+0000 7fae4fc30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:34:41.940 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:41.941+0000 7f21f5c43040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:34:41.993 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:41.993+0000 7fae4fc30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:34:41.996 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:41.997+0000 7f21f5c43040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:34:42.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.153+0000 7f48fffff640 1 -- 172.21.15.47:0/3442055015 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f4910005810 con 0x7f49180ad1b0 2024-03-25T19:34:42.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.153+0000 7f48fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:34:42.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.153+0000 7f48fffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:34:42.152 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 10 librados: watch_flush enter 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 10 librados: watch_flush exit 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 10 client.4111.objecter successfully canceled tick 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 10 monclient: shutdown 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:42.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 >> v1:172.21.15.142:6789/0 conn(0x7f49180ad1b0 legacy=0x7f49181db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:42.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 shutdown_connections 2024-03-25T19:34:42.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 >> 172.21.15.47:0/3442055015 conn(0x7f491806b560 msgr2=0x7f49180b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:42.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 shutdown_connections 2024-03-25T19:34:42.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 1 -- 172.21.15.47:0/3442055015 wait complete. 2024-03-25T19:34:42.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:42.154+0000 7f491dc1e640 1 librados: shutdown 2024-03-25T19:34:42.168 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:34:42.168 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:34:42.169 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:42.169 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:34:42.198 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:42.198 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:34:42.265 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:42.265 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:34:42.332 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-25T19:34:42.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:34:42.399 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:42.399 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:34:42.426 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:42.426 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:34:42.494 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:42.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:34:42.549 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.549+0000 7fae4fc30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:34:42.560 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-25T19:34:42.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:34:42.595 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:42.595+0000 7f21f5c43040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:34:42.605 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.606+0000 7fae4fc30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:34:42.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 24d0b8fc-9364-43bb-9d61-e0b8db08e324 0 2024-03-25T19:34:42.650 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:42.651+0000 7f21f5c43040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:34:42.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.653+0000 7fae4fc30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:34:42.696 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:42.697+0000 7f21f5c43040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:34:42.709 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.710+0000 7fae4fc30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient: get_monmap_and_config 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient: build_initial_monmap 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient: monmap: 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:42.753577+0000 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:42.753577+0000 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:42.752 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:42.753 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:42.753 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:42.753 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.753 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:42.753+0000 7f21f5c43040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 1 Processor -- start 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 1 -- start start 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient: init 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient(hunting): picked mon.noname-c con 0x7f8af00a9920 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient(hunting): picked mon.noname-a con 0x7f8af00b45a0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient(hunting): picked mon.noname-b con 0x7f8af0146450 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af006b100 con 0x7f8af00b45a0 2024-03-25T19:34:42.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af00b2940 con 0x7f8af00a9920 2024-03-25T19:34:42.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af013fec0 con 0x7f8af0146450 2024-03-25T19:34:42.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:42.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.753+0000 7f8af7bf6640 10 monclient(hunting): authenticate will time out at 5603.061961s 2024-03-25T19:34:42.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.754+0000 7f8af5992640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8af00a9920 0x7f8af013f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47650/0 (socket says 172.21.15.142:47650) 2024-03-25T19:34:42.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.754+0000 7f8af5992640 1 -- 172.21.15.142:0/1715433574 learned_addr learned my addr 172.21.15.142:0/1715433574 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:42.756 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.755+0000 7fae4fc30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.756+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384442460 0 0) 0x7f8af013fec0 con 0x7f8af0146450 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.756+0000 7f8af4990640 10 monclient(hunting): _negotiate 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.756+0000 7f8af4990640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.756+0000 7f8af4990640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.756+0000 7f8af4990640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.756+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc003520 con 0x7f8af0146450 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.757+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026333001 0 0) 0x7f8af00b2940 con 0x7f8af00a9920 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.757+0000 7f8af4990640 10 monclient(hunting): _negotiate 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.757+0000 7f8af4990640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.757+0000 7f8af4990640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.757+0000 7f8af4990640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:34:42.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.757+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af013fec0 con 0x7f8af00a9920 2024-03-25T19:34:42.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.758+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404061636 0 0) 0x7f8af006b100 con 0x7f8af00b45a0 2024-03-25T19:34:42.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.758+0000 7f8af4990640 10 monclient(hunting): _negotiate 2024-03-25T19:34:42.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.758+0000 7f8af4990640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:42.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.758+0000 7f8af4990640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:42.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.758+0000 7f8af4990640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:34:42.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.758+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af00b2940 con 0x7f8af00b45a0 2024-03-25T19:34:42.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074709954 0 0) 0x7f8af013fec0 con 0x7f8af00a9920 2024-03-25T19:34:42.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 >> v1:172.21.15.47:6790/0 conn(0x7f8af0146450 legacy=0x7f8af014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:42.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 >> v1:172.21.15.47:6789/0 conn(0x7f8af00b45a0 legacy=0x7f8af0145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:42.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: _finish_hunting 0 2024-03-25T19:34:42.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 1 monclient: found mon.noname-c 2024-03-25T19:34:42.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0140050 con 0x7f8af00a9920 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: _finish_auth 0 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: _check_auth_tickets 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af7bf6640 5 monclient: authenticate success, global_id 4114 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af7bf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aec001490 con 0x7f8af00a9920 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af4990640 10 monclient: dump: 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:42.761 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.760+0000 7f8af7bf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:42.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aec0016e0 con 0x7f8af00a9920 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af4990640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af4990640 1 -- 172.21.15.142:0/1715433574 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aec001a70 con 0x7f8af00a9920 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 4 set_mon_vals no callback set 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af4990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af4990640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af4990640 10 monclient: dump: 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.766 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 10 monclient: shutdown 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1715433574 >> v1:172.21.15.142:6789/0 conn(0x7f8af00a9920 legacy=0x7f8af013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1715433574 shutdown_connections 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.762+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1715433574 >> 172.21.15.142:0/1715433574 conn(0x7f8af00b7f40 msgr2=0x7f8af00b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1715433574 shutdown_connections 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1715433574 wait complete. 2024-03-25T19:34:42.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 10 monclient: build_initial_monmap 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 10 monclient: monmap: 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 1 librados: starting msgr at 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 1 librados: starting objecter 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.763+0000 7f8af7bf6640 1 Processor -- start 2024-03-25T19:34:42.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 1 -- start start 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 1 librados: setting wanted keys 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 1 librados: calling monclient init 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient: init 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient(hunting): picked mon.noname-a con 0x7f8af00a9920 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient(hunting): picked mon.noname-c con 0x7f8af00b45a0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient(hunting): picked mon.noname-b con 0x7f8af0146450 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af0142ab0 con 0x7f8af00a9920 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af0137b40 con 0x7f8af0146450 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af01d4f90 con 0x7f8af00b45a0 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.764+0000 7f8af7bf6640 10 monclient(hunting): authenticate will time out at 5603.072777s 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8af5992640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8af00a9920 0x7f8af01d7450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.142:60922/0 (socket says 172.21.15.142:60922) 2024-03-25T19:34:42.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8af5992640 1 -- 172.21.15.142:0/1257914845 learned_addr learned my addr 172.21.15.142:0/1257914845 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:42.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8ae67fc640 10 client.?.objecter ms_handle_connect 0x7f8af0146450 2024-03-25T19:34:42.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8ae67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:42.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8ae67fc640 10 client.?.objecter ms_handle_connect 0x7f8af00b45a0 2024-03-25T19:34:42.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8ae67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:42.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8ae67fc640 10 client.?.objecter ms_handle_connect 0x7f8af00a9920 2024-03-25T19:34:42.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.765+0000 7f8ae67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062945455 0 0) 0x7f8af01d4f90 con 0x7f8af00b45a0 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad4004eb0 con 0x7f8af00b45a0 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424278239 0 0) 0x7f8af0137b40 con 0x7f8af0146450 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:42.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.767+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af01d4f90 con 0x7f8af0146450 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.768+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412395204 0 0) 0x7f8af0142ab0 con 0x7f8af00a9920 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.768+0000 7f8ae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.768+0000 7f8ae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.768+0000 7f8ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.768+0000 7f8ae67fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:34:42.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.768+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af0137b40 con 0x7f8af00a9920 2024-03-25T19:34:42.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1520299677 0 0) 0x7f8af01d4f90 con 0x7f8af0146450 2024-03-25T19:34:42.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af0142ab0 con 0x7f8af0146450 2024-03-25T19:34:42.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8ae8001a20 con 0x7f8af0146450 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): dump: 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270457510 0 0) 0x7f8af0137b40 con 0x7f8af00a9920 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af01d4f90 con 0x7f8af00a9920 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aec001550 con 0x7f8af00a9920 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.774+0000 7f8ae67fc640 10 monclient(hunting): dump: 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:42.776 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:42.777 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:42.777 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4134371256 0 0) 0x7f8af01d4f90 con 0x7f8af00a9920 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 >> v1:172.21.15.47:6790/0 conn(0x7f8af00b45a0 legacy=0x7f8af0136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 >> v1:172.21.15.142:6789/0 conn(0x7f8af0146450 legacy=0x7f8af0137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 1 monclient: found mon.a 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af01d7b60 con 0x7f8af00a9920 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:42.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 10 monclient: _renew_subs 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0142680 con 0x7f8af00a9920 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 10 monclient: _renew_subs 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af01423a0 con 0x7f8af00a9920 2024-03-25T19:34:42.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 1 librados: init done 2024-03-25T19:34:42.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.779+0000 7f8af7bf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:42.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.780+0000 7f8af7bf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:42.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.780+0000 7f8af7bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:42.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.780+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af004f730 con 0x7f8af00a9920 2024-03-25T19:34:42.816 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:42.816+0000 7f21f5c43040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:34:42.875 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.875+0000 7fae4fc30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.914+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aec004a90 con 0x7f8af00a9920 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.914+0000 7f8ae67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.914+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aec004da0 con 0x7f8af00a9920 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.914+0000 7f8ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 10 monclient: dump: 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:42.915 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:42.916 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:42.916 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:42.916 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:42.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.914+0000 7f8ae7fff640 4 set_mon_vals no callback set 2024-03-25T19:34:42.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8aec005290 con 0x7f8af00a9920 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1456055181 0 0) 0x7f8aec0067b0 con 0x7f8af00a9920 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 10 client.4119.objecter ms_dispatch 0x7f8af0000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.915+0000 7f8ae67fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.916+0000 7f8ae67fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.916+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8aec006a20 con 0x7f8af00a9920 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.916+0000 7f8ae67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:42.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.916+0000 7f8ae67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:42.935 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:42.936+0000 7f21f5c43040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:34:42.945 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.945+0000 7fae4fc30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:34:42.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.988+0000 7f8af7bf6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "24d0b8fc-9364-43bb-9d61-e0b8db08e324", "id": 0}] 2024-03-25T19:34:42.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.988+0000 7f8af7bf6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "24d0b8fc-9364-43bb-9d61-e0b8db08e324", "id": 0}] 2024-03-25T19:34:42.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.988+0000 7f8af7bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:42.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:42.988+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd new", "uuid": "24d0b8fc-9364-43bb-9d61-e0b8db08e324", "id": 0} v 0) v1 -- 0x7f8af0142870 con 0x7f8af00a9920 2024-03-25T19:34:42.992 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:42.993+0000 7fae4fc30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:34:43.008 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.009+0000 7f21f5c43040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:34:43.045 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.045+0000 7fae4fc30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:34:43.056 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.057+0000 7f21f5c43040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:34:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.076+0000 7f8ae67fc640 1 -- 172.21.15.142:0/1257914845 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24d0b8fc-9364-43bb-9d61-e0b8db08e324", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3335492250 0 732707515) 0x7f8aec0057d0 con 0x7f8af00a9920 2024-03-25T19:34:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.076+0000 7f8ae67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "24d0b8fc-9364-43bb-9d61-e0b8db08e324", "id": 0}] 2024-03-25T19:34:43.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.076+0000 7f8ae67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:43.076 INFO:teuthology.orchestra.run.smithi142.stdout:0 2024-03-25T19:34:43.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.076+0000 7f8af7bf6640 10 librados: watch_flush enter 2024-03-25T19:34:43.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.076+0000 7f8af7bf6640 10 librados: watch_flush exit 2024-03-25T19:34:43.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:34:43.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:34:43.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 10 monclient: shutdown 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 >> v1:172.21.15.47:6789/0 conn(0x7f8af00a9920 legacy=0x7f8af01d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 shutdown_connections 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 >> 172.21.15.142:0/1257914845 conn(0x7f8af00b7f40 msgr2=0x7f8af0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 shutdown_connections 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 1 -- 172.21.15.142:0/1257914845 wait complete. 2024-03-25T19:34:43.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.077+0000 7f8af7bf6640 1 librados: shutdown 2024-03-25T19:34:43.089 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 27159e00-50f8-4194-9d01-5726783de16d 1 2024-03-25T19:34:43.102 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.102+0000 7fae4fc30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:34:43.113 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.114+0000 7f21f5c43040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:34:43.164 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.165+0000 7f21f5c43040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:34:43.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.190+0000 7f45c79e8640 10 monclient: get_monmap_and_config 2024-03-25T19:34:43.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.190+0000 7f45c79e8640 10 monclient: build_initial_monmap 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.190+0000 7f45c79e8640 10 monclient: monmap: 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:43.192162+0000 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:43.192162+0000 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:43.191 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.191+0000 7f45c79e8640 1 Processor -- start 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.191+0000 7f45c79e8640 1 -- start start 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.191+0000 7f45c79e8640 10 monclient: init 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient(hunting): picked mon.noname-b con 0x7f45c00b4700 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient(hunting): picked mon.noname-a con 0x7f45c0142a20 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient(hunting): picked mon.noname-c con 0x7f45c00a96b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c00a9d60 con 0x7f45c0142a20 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c013ff10 con 0x7f45c00a96b0 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c00aa1f0 con 0x7f45c00b4700 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c79e8640 10 monclient(hunting): authenticate will time out at 5603.500497s 2024-03-25T19:34:43.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c5f85640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f45c00a96b0 0x7f45c013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47674/0 (socket says 172.21.15.142:47674) 2024-03-25T19:34:43.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.192+0000 7f45c5f85640 1 -- 172.21.15.142:0/722849036 learned_addr learned my addr 172.21.15.142:0/722849036 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557323015 0 0) 0x7f45c00aa1f0 con 0x7f45c00b4700 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ac003520 con 0x7f45c00b4700 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459356811 0 0) 0x7f45c013ff10 con 0x7f45c00a96b0 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:34:43.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.195+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c00aa1f0 con 0x7f45c00a96b0 2024-03-25T19:34:43.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2598776473 0 0) 0x7f45ac003520 con 0x7f45c00b4700 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 >> v1:172.21.15.142:6789/0 conn(0x7f45c00a96b0 legacy=0x7f45c013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 >> v1:172.21.15.47:6789/0 conn(0x7f45c0142a20 legacy=0x7f45c013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 1 monclient: found mon.noname-b 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00b4ae0 con 0x7f45c00b4700 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f45bc001690 con 0x7f45c00b4700 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45c79e8640 5 monclient: authenticate success, global_id 4115 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45c79e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:43.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45b7fff640 10 monclient: dump: 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.198+0000 7f45c79e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45bc0017f0 con 0x7f45c00b4700 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45b7fff640 1 -- 172.21.15.142:0/722849036 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f45bc001c00 con 0x7f45c00b4700 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45b7fff640 10 monclient: dump: 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.225 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45c79e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:43.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.225+0000 7f45c79e8640 4 set_mon_vals no callback set 2024-03-25T19:34:43.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 10 monclient: shutdown 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 1 -- 172.21.15.142:0/722849036 >> v1:172.21.15.47:6790/0 conn(0x7f45c00b4700 legacy=0x7f45c0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 1 -- 172.21.15.142:0/722849036 shutdown_connections 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 1 -- 172.21.15.142:0/722849036 >> 172.21.15.142:0/722849036 conn(0x7f45c00668e0 msgr2=0x7f45c0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 1 -- 172.21.15.142:0/722849036 shutdown_connections 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 1 -- 172.21.15.142:0/722849036 wait complete. 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 10 monclient: build_initial_monmap 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.226+0000 7f45c79e8640 10 monclient: monmap: 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:43.229 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 librados: starting msgr at 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 librados: starting objecter 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 Processor -- start 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 -- start start 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 librados: setting wanted keys 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 librados: calling monclient init 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient: init 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient(hunting): picked mon.noname-a con 0x7f45c00a96b0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient(hunting): picked mon.noname-c con 0x7f45c00b4700 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient(hunting): picked mon.noname-b con 0x7f45c0142a20 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c0139550 con 0x7f45c00a96b0 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c01397c0 con 0x7f45c0142a20 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c0139aa0 con 0x7f45c00b4700 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.227+0000 7f45c79e8640 10 monclient(hunting): authenticate will time out at 5603.536032s 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.228+0000 7f45c5f85640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f45c0142a20 0x7f45c01381e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47680/0 (socket says 172.21.15.142:47680) 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.228+0000 7f45c5f85640 1 -- 172.21.15.142:0/2601713470 learned_addr learned my addr 172.21.15.142:0/2601713470 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.228+0000 7f45b5ffb640 10 client.?.objecter ms_handle_connect 0x7f45c00a96b0 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.228+0000 7f45b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.228+0000 7f45b5ffb640 10 client.?.objecter ms_handle_connect 0x7f45c0142a20 2024-03-25T19:34:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.228+0000 7f45b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.230+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531097415 0 0) 0x7f45c0139550 con 0x7f45c00a96b0 2024-03-25T19:34:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.230+0000 7f45b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.230+0000 7f45b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.230+0000 7f45b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.230+0000 7f45b5ffb640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:34:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.230+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a4004eb0 con 0x7f45c00a96b0 2024-03-25T19:34:43.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.235+0000 7f45b5ffb640 10 client.?.objecter ms_handle_connect 0x7f45c00b4700 2024-03-25T19:34:43.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.235+0000 7f45b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682196386 0 0) 0x7f45a4004eb0 con 0x7f45c00a96b0 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45c0139550 con 0x7f45c00a96b0 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f45bc001bc0 con 0x7f45c00a96b0 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.236 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.236+0000 7f45b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 81061886 0 0) 0x7f45c0139550 con 0x7f45c00a96b0 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 >> v1:172.21.15.47:6790/0 conn(0x7f45c00b4700 legacy=0x7f45c0139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 >> v1:172.21.15.142:6789/0 conn(0x7f45c0142a20 legacy=0x7f45c01381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 1 monclient: found mon.a 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c01375c0 con 0x7f45c00a96b0 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:43.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 10 monclient: _renew_subs 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c01388f0 con 0x7f45c00a96b0 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 10 monclient: _renew_subs 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c01e3690 con 0x7f45c00a96b0 2024-03-25T19:34:43.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.240+0000 7f45c79e8640 1 librados: init done 2024-03-25T19:34:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.241+0000 7f45c79e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.241+0000 7f45c79e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.241+0000 7f45c79e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.241+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c004f730 con 0x7f45c00a96b0 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45bc0038a0 con 0x7f45c00a96b0 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f45bc003bb0 con 0x7f45c00a96b0 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b5ffb640 10 monclient: dump: 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.262+0000 7f45b77fe640 4 set_mon_vals no callback set 2024-03-25T19:34:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.401+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f45bc0040a0 con 0x7f45c00a96b0 2024-03-25T19:34:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.401+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1463935243 0 0) 0x7f45bc005590 con 0x7f45c00a96b0 2024-03-25T19:34:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.401+0000 7f45b5ffb640 10 client.4125.objecter ms_dispatch 0x7f45c0000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:34:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.401+0000 7f45b5ffb640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:34:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.401+0000 7f45b5ffb640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:34:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.401+0000 7f45b5ffb640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:34:43.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.402+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f45bc005850 con 0x7f45c00a96b0 2024-03-25T19:34:43.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.402+0000 7f45b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.402+0000 7f45b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:43.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.475+0000 7f45c79e8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27159e00-50f8-4194-9d01-5726783de16d", "id": 1}] 2024-03-25T19:34:43.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.475+0000 7f45c79e8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27159e00-50f8-4194-9d01-5726783de16d", "id": 1}] 2024-03-25T19:34:43.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.475+0000 7f45c79e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:43.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.475+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd new", "uuid": "27159e00-50f8-4194-9d01-5726783de16d", "id": 1} v 0) v1 -- 0x7f45c00b2510 con 0x7f45c00a96b0 2024-03-25T19:34:43.501 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.502+0000 7fae4fc30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:34:43.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.544+0000 7f45b5ffb640 1 -- 172.21.15.142:0/2601713470 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27159e00-50f8-4194-9d01-5726783de16d", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4177990183 0 940483276) 0x7f45bc005ad0 con 0x7f45c00a96b0 2024-03-25T19:34:43.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.544+0000 7f45b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27159e00-50f8-4194-9d01-5726783de16d", "id": 1}] 2024-03-25T19:34:43.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.544+0000 7f45b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:43.546 INFO:teuthology.orchestra.run.smithi142.stdout:1 2024-03-25T19:34:43.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 10 librados: watch_flush enter 2024-03-25T19:34:43.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 10 librados: watch_flush exit 2024-03-25T19:34:43.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:34:43.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:34:43.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 10 monclient: shutdown 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 >> v1:172.21.15.47:6789/0 conn(0x7f45c00a96b0 legacy=0x7f45c0135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 shutdown_connections 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 >> 172.21.15.142:0/2601713470 conn(0x7f45c00668e0 msgr2=0x7f45c00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 shutdown_connections 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 1 -- 172.21.15.142:0/2601713470 wait complete. 2024-03-25T19:34:43.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.545+0000 7f45c79e8640 1 librados: shutdown 2024-03-25T19:34:43.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 3f06d608-135b-4afa-a600-7d20cd766e57 2 2024-03-25T19:34:43.607 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.607+0000 7f21f5c43040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:34:43.644 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.646+0000 7fae4fc30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:34:43.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.695+0000 7f922d4ba640 10 monclient: get_monmap_and_config 2024-03-25T19:34:43.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.695+0000 7f922d4ba640 10 monclient: build_initial_monmap 2024-03-25T19:34:43.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.695+0000 7f922d4ba640 10 monclient: monmap: 2024-03-25T19:34:43.694 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:43.694 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:43.695940+0000 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:43.695940+0000 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:43.695 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.696+0000 7f922d4ba640 1 Processor -- start 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.696+0000 7f922d4ba640 1 -- start start 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.696+0000 7f922d4ba640 10 monclient: init 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f92280a9920 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f92280b45a0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9228146450 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922806b100 con 0x7f92280b45a0 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280b2940 con 0x7f9228146450 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922813fec0 con 0x7f92280a9920 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:43.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f922d4ba640 10 monclient(hunting): authenticate will time out at 5604.004278s 2024-03-25T19:34:43.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f92277fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9228146450 0x7f922814dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47684/0 (socket says 172.21.15.142:47684) 2024-03-25T19:34:43.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.697+0000 7f92277fe640 1 -- 172.21.15.142:0/1540149934 learned_addr learned my addr 172.21.15.142:0/1540149934 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354250330 0 0) 0x7f92280b2940 con 0x7f9228146450 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9214003520 con 0x7f9228146450 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117467791 0 0) 0x7f922806b100 con 0x7f92280b45a0 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:34:43.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.700+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92280b2940 con 0x7f92280b45a0 2024-03-25T19:34:43.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.701+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608770990 0 0) 0x7f922813fec0 con 0x7f92280a9920 2024-03-25T19:34:43.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.701+0000 7f92267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.701+0000 7f92267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.701+0000 7f92267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.701+0000 7f92267fc640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:34:43.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.701+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f922806b100 con 0x7f92280a9920 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012191606 0 0) 0x7f9214003520 con 0x7f9228146450 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 >> v1:172.21.15.47:6790/0 conn(0x7f92280a9920 legacy=0x7f922813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 >> v1:172.21.15.47:6789/0 conn(0x7f92280b45a0 legacy=0x7f9228145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 1 monclient: found mon.noname-c 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:43.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228140050 con 0x7f9228146450 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9220001a10 con 0x7f9228146450 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f92267fc640 10 monclient: dump: 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.703 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.704 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.704 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f922d4ba640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:34:43.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.703+0000 7f922d4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9220001b70 con 0x7f9228146450 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f92267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f92267fc640 1 -- 172.21.15.142:0/1540149934 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9220001de0 con 0x7f9228146450 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 4 set_mon_vals no callback set 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f92267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f92267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:43.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f92267fc640 10 monclient: dump: 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 10 monclient: shutdown 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:43.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 1 -- 172.21.15.142:0/1540149934 >> v1:172.21.15.142:6789/0 conn(0x7f9228146450 legacy=0x7f922814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 1 -- 172.21.15.142:0/1540149934 shutdown_connections 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 1 -- 172.21.15.142:0/1540149934 >> 172.21.15.142:0/1540149934 conn(0x7f92280b7f40 msgr2=0x7f92280b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 1 -- 172.21.15.142:0/1540149934 shutdown_connections 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.705+0000 7f922d4ba640 1 -- 172.21.15.142:0/1540149934 wait complete. 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.706+0000 7f922d4ba640 10 monclient: build_initial_monmap 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.706+0000 7f922d4ba640 10 monclient: monmap: 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:43.709 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.706+0000 7f922d4ba640 1 librados: starting msgr at 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.706+0000 7f922d4ba640 1 librados: starting objecter 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.706+0000 7f922d4ba640 1 Processor -- start 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 1 -- start start 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 1 librados: setting wanted keys 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 1 librados: calling monclient init 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient: init 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f92280a9920 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:43.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f92280b45a0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9228146450 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9228141a70 con 0x7f92280b45a0 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9228143240 con 0x7f92280a9920 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922813b570 con 0x7f9228146450 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.707+0000 7f922d4ba640 10 monclient(hunting): authenticate will time out at 5604.014739s 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f9226ffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f92280a9920 0x7f922813c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47686/0 (socket says 172.21.15.142:47686) 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f9226ffd640 1 -- 172.21.15.142:0/1235011904 learned_addr learned my addr 172.21.15.142:0/1235011904 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f921f7fe640 10 client.?.objecter ms_handle_connect 0x7f92280a9920 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f921f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f921f7fe640 10 client.?.objecter ms_handle_connect 0x7f9228146450 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f921f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f921f7fe640 10 client.?.objecter ms_handle_connect 0x7f92280b45a0 2024-03-25T19:34:43.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.708+0000 7f921f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109698504 0 0) 0x7f9228143240 con 0x7f92280a9920 2024-03-25T19:34:43.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208004eb0 con 0x7f92280a9920 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130386245 0 0) 0x7f92280b2940 con 0x7f92280b45a0 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.710+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9228143240 con 0x7f92280b45a0 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.711+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974257504 0 0) 0x7f922813b570 con 0x7f9228146450 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.711+0000 7f921f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.711+0000 7f921f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.711+0000 7f921f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.711+0000 7f921f7fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:34:43.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.711+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92280b2940 con 0x7f9228146450 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.714+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3721752612 0 0) 0x7f9208004eb0 con 0x7f92280a9920 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.714+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f922813b570 con 0x7f92280a9920 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2263536030 0 0) 0x7f9228143240 con 0x7f92280b45a0 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9208004eb0 con 0x7f92280b45a0 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9210001810 con 0x7f92280b45a0 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:43.715 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.715+0000 7f921f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.716+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226413843 0 0) 0x7f92280b2940 con 0x7f9228146450 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.716+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9228143240 con 0x7f9228146450 2024-03-25T19:34:43.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.716+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9220001e80 con 0x7f9228146450 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.716+0000 7f921f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.716+0000 7f921f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.716+0000 7f921f7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.717 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9218001ad0 con 0x7f92280a9920 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.719 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 596417672 0 0) 0x7f922813b570 con 0x7f92280a9920 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 >> v1:172.21.15.47:6790/0 conn(0x7f9228146450 legacy=0x7f92281e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 >> v1:172.21.15.47:6789/0 conn(0x7f92280b45a0 legacy=0x7f92281d78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 1 monclient: found mon.b 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922813b7e0 con 0x7f92280a9920 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f921f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 5 monclient: authenticate success, global_id 4126 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 10 monclient: _renew_subs 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:43.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922813bac0 con 0x7f92280a9920 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 10 monclient: _renew_subs 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f922813a8b0 con 0x7f92280a9920 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.718+0000 7f922d4ba640 1 librados: init done 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.719+0000 7f922d4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.719+0000 7f922d4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.719+0000 7f922d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.719+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92280b7db0 con 0x7f92280a9920 2024-03-25T19:34:43.751 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.752+0000 7f21f5c43040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:34:43.758 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.759+0000 7fae4fc30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:34:43.805 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.806+0000 7fae4fc30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92180017a0 con 0x7f92280a9920 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f92180038c0 con 0x7f92280a9920 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 monclient: dump: 2024-03-25T19:34:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9218003e70 con 0x7f92280a9920 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f9225ffb640 4 set_mon_vals no callback set 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 674642194 0 0) 0x7f9218006630 con 0x7f92280a9920 2024-03-25T19:34:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 client.4126.objecter ms_dispatch 0x7f9228000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:34:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:34:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:34:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T19:34:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9218006a50 con 0x7f92280a9920 2024-03-25T19:34:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.818+0000 7f921f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:43.883 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.883+0000 7f21f5c43040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:34:43.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.890+0000 7f922d4ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3f06d608-135b-4afa-a600-7d20cd766e57", "id": 2}] 2024-03-25T19:34:43.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.890+0000 7f922d4ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3f06d608-135b-4afa-a600-7d20cd766e57", "id": 2}] 2024-03-25T19:34:43.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.890+0000 7f922d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:43.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.890+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3f06d608-135b-4afa-a600-7d20cd766e57", "id": 2} v 0) v1 -- 0x7f9228143020 con 0x7f92280a9920 2024-03-25T19:34:43.922 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:43.923+0000 7fae4fc30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:34:43.935 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:43.935+0000 7f21f5c43040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:34:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.969+0000 7f921f7fe640 1 -- 172.21.15.142:0/1235011904 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3f06d608-135b-4afa-a600-7d20cd766e57", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 628061837 0 216599125) 0x7f9218004470 con 0x7f92280a9920 2024-03-25T19:34:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.969+0000 7f921f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3f06d608-135b-4afa-a600-7d20cd766e57", "id": 2}] 2024-03-25T19:34:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.969+0000 7f921f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:43.969 INFO:teuthology.orchestra.run.smithi142.stdout:2 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 10 librados: watch_flush enter 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 10 librados: watch_flush exit 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 10 client.4126.objecter successfully canceled tick 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 10 monclient: shutdown 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 >> v1:172.21.15.142:6789/0 conn(0x7f92280a9920 legacy=0x7f922813c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 shutdown_connections 2024-03-25T19:34:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 >> 172.21.15.142:0/1235011904 conn(0x7f92280b7f40 msgr2=0x7f9228074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 shutdown_connections 2024-03-25T19:34:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 1 -- 172.21.15.142:0/1235011904 wait complete. 2024-03-25T19:34:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:43.970+0000 7f922d4ba640 1 librados: shutdown 2024-03-25T19:34:43.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 6457e632-00d5-4e26-9170-109d2d2fd4e5 3 2024-03-25T19:34:44.025 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:44.026+0000 7fae4fc30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:34:44.061 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:44.062+0000 7f21f5c43040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.079+0000 7ff8437f4640 10 monclient: get_monmap_and_config 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.079+0000 7ff8437f4640 10 monclient: build_initial_monmap 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.079+0000 7ff8437f4640 10 monclient: monmap: 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:44.079873+0000 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:44.079873+0000 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:44.078 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.079 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:44.079 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:44.079 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:44.079 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.080+0000 7ff8437f4640 1 Processor -- start 2024-03-25T19:34:44.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.080+0000 7ff8437f4640 1 -- start start 2024-03-25T19:34:44.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.080+0000 7ff8437f4640 10 monclient: init 2024-03-25T19:34:44.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.080+0000 7ff8437f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.080+0000 7ff8437f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff83c0b0530 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff83c0b0910 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff83c142800 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0719f0 con 0x7ff83c0b0530 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c070980 con 0x7ff83c0b0910 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c072940 con 0x7ff83c142800 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff8437f4640 10 monclient(hunting): authenticate will time out at 5604.388262s 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff840d8f640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff83c0b0910 0x7ff83c071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47700/0 (socket says 172.21.15.142:47700) 2024-03-25T19:34:44.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.081+0000 7ff840d8f640 1 -- 172.21.15.142:0/2770122170 learned_addr learned my addr 172.21.15.142:0/2770122170 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:44.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927045059 0 0) 0x7ff83c070980 con 0x7ff83c0b0910 2024-03-25T19:34:44.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:34:44.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff828003520 con 0x7ff83c0b0910 2024-03-25T19:34:44.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954264434 0 0) 0x7ff83c072940 con 0x7ff83c142800 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c070980 con 0x7ff83c142800 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354546915 0 0) 0x7ff83c0719f0 con 0x7ff83c0b0530 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:34:44.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.084+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c072940 con 0x7ff83c0b0530 2024-03-25T19:34:44.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625519226 0 0) 0x7ff828003520 con 0x7ff83c0b0910 2024-03-25T19:34:44.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 >> v1:172.21.15.47:6790/0 conn(0x7ff83c142800 legacy=0x7ff83c142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 >> v1:172.21.15.47:6789/0 conn(0x7ff83c0b0530 legacy=0x7ff83c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:44.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 1 monclient: found mon.noname-c 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c072520 con 0x7ff83c0b0910 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff8437f4640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff8437f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff82c001650 con 0x7ff83c0b0910 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff833fff640 10 monclient: dump: 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.087 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.087+0000 7ff8437f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff82c0018a0 con 0x7ff83c0b0910 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff833fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff833fff640 1 -- 172.21.15.142:0/2770122170 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff82c001c30 con 0x7ff83c0b0910 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff8437f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff8437f4640 4 set_mon_vals no callback set 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff833fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.088+0000 7ff833fff640 10 monclient: dump: 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.091 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 10 monclient: shutdown 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 1 -- 172.21.15.142:0/2770122170 >> v1:172.21.15.142:6789/0 conn(0x7ff83c0b0910 legacy=0x7ff83c071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 1 -- 172.21.15.142:0/2770122170 shutdown_connections 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 1 -- 172.21.15.142:0/2770122170 >> 172.21.15.142:0/2770122170 conn(0x7ff83c06bcb0 msgr2=0x7ff83c06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 1 -- 172.21.15.142:0/2770122170 shutdown_connections 2024-03-25T19:34:44.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.089+0000 7ff8437f4640 1 -- 172.21.15.142:0/2770122170 wait complete. 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 10 monclient: build_initial_monmap 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 10 monclient: monmap: 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 1 librados: starting msgr at 2024-03-25T19:34:44.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 1 librados: starting objecter 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 1 Processor -- start 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 1 -- start start 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 1 librados: setting wanted keys 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 1 librados: calling monclient init 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.090+0000 7ff8437f4640 10 monclient: init 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff83c0b0530 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff83c142800 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff83c13e4c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c1d72b0 con 0x7ff83c142800 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c136590 con 0x7ff83c0b0530 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c136720 con 0x7ff83c13e4c0 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.091+0000 7ff8437f4640 10 monclient(hunting): authenticate will time out at 5604.398739s 2024-03-25T19:34:44.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff841590640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff83c0b0530 0x7ff83c1d6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47714/0 (socket says 172.21.15.142:47714) 2024-03-25T19:34:44.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff841590640 1 -- 172.21.15.142:0/1535860425 learned_addr learned my addr 172.21.15.142:0/1535860425 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:44.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff831ffb640 10 client.?.objecter ms_handle_connect 0x7ff83c0b0530 2024-03-25T19:34:44.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff831ffb640 10 client.?.objecter ms_handle_connect 0x7ff83c13e4c0 2024-03-25T19:34:44.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff831ffb640 10 client.?.objecter ms_handle_connect 0x7ff83c142800 2024-03-25T19:34:44.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.092+0000 7ff831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383263033 0 0) 0x7ff83c136590 con 0x7ff83c0b0530 2024-03-25T19:34:44.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff820004eb0 con 0x7ff83c0b0530 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451137678 0 0) 0x7ff83c136720 con 0x7ff83c13e4c0 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.094+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c136590 con 0x7ff83c13e4c0 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.095+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507649869 0 0) 0x7ff83c1d72b0 con 0x7ff83c142800 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.095+0000 7ff831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.095+0000 7ff831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.095+0000 7ff831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.095+0000 7ff831ffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:34:44.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.095+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c136720 con 0x7ff83c142800 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2320610042 0 0) 0x7ff820004eb0 con 0x7ff83c0b0530 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff83c1d72b0 con 0x7ff83c0b0530 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff838001a10 con 0x7ff83c0b0530 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:44.098 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.099+0000 7ff831ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.101+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 685952282 0 0) 0x7ff83c136590 con 0x7ff83c13e4c0 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.101+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff820004eb0 con 0x7ff83c13e4c0 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.101+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff834001a90 con 0x7ff83c13e4c0 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.101+0000 7ff831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.101+0000 7ff831ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.101+0000 7ff831ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.100 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.101 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.101 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.101 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.101 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857964997 0 0) 0x7ff83c136720 con 0x7ff83c142800 2024-03-25T19:34:44.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff83c136590 con 0x7ff83c142800 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff82c001a10 con 0x7ff83c142800 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.103 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 811214397 0 0) 0x7ff83c1d72b0 con 0x7ff83c0b0530 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 >> v1:172.21.15.47:6790/0 conn(0x7ff83c13e4c0 legacy=0x7ff83c13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 >> v1:172.21.15.47:6789/0 conn(0x7ff83c142800 legacy=0x7ff83c135e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 monclient: found mon.b 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c1368b0 con 0x7ff83c0b0530 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 10 monclient: _renew_subs 2024-03-25T19:34:44.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c0af650 con 0x7ff83c0b0530 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 10 monclient: _renew_subs 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c13d7d0 con 0x7ff83c0b0530 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.102+0000 7ff8437f4640 1 librados: init done 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.103+0000 7ff8437f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.103+0000 7ff8437f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.103+0000 7ff8437f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.103+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c04f730 con 0x7ff83c0b0530 2024-03-25T19:34:44.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff838001c70 con 0x7ff83c0b0530 2024-03-25T19:34:44.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:44.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff838003a60 con 0x7ff83c0b0530 2024-03-25T19:34:44.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 10 monclient: dump: 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff838004010 con 0x7ff83c0b0530 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff8337fe640 4 set_mon_vals no callback set 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 895635029 0 0) 0x7ff838006750 con 0x7ff83c0b0530 2024-03-25T19:34:44.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 10 client.4132.objecter ms_dispatch 0x7ff83c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:34:44.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:34:44.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:34:44.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.106+0000 7ff831ffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T19:34:44.163 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:44.163+0000 7f21f5c43040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:34:44.187 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:44.188+0000 7fae4fc30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:34:44.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.205+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff8380042c0 con 0x7ff83c0b0530 2024-03-25T19:34:44.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.205+0000 7ff831ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.205+0000 7ff831ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:44.233 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:44.234+0000 7fae4fc30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:34:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.278+0000 7ff8437f4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6457e632-00d5-4e26-9170-109d2d2fd4e5", "id": 3}] 2024-03-25T19:34:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.278+0000 7ff8437f4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6457e632-00d5-4e26-9170-109d2d2fd4e5", "id": 3}] 2024-03-25T19:34:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.278+0000 7ff8437f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.278+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6457e632-00d5-4e26-9170-109d2d2fd4e5", "id": 3} v 0) v1 -- 0x7ff83c1d7440 con 0x7ff83c0b0530 2024-03-25T19:34:44.329 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:44.329+0000 7f21f5c43040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:34:44.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.361+0000 7ff831ffb640 1 -- 172.21.15.142:0/1535860425 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6457e632-00d5-4e26-9170-109d2d2fd4e5", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3818083530 0 525046306) 0x7ff838006370 con 0x7ff83c0b0530 2024-03-25T19:34:44.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.361+0000 7ff831ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6457e632-00d5-4e26-9170-109d2d2fd4e5", "id": 3}] 2024-03-25T19:34:44.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.361+0000 7ff831ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:44.360 INFO:teuthology.orchestra.run.smithi142.stdout:3 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 10 librados: watch_flush enter 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 10 librados: watch_flush exit 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 10 client.4132.objecter successfully canceled tick 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 10 monclient: shutdown 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 >> v1:172.21.15.142:6789/0 conn(0x7ff83c0b0530 legacy=0x7ff83c1d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 shutdown_connections 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 >> 172.21.15.142:0/1535860425 conn(0x7ff83c06bcb0 msgr2=0x7ff83c0b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 shutdown_connections 2024-03-25T19:34:44.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 1 -- 172.21.15.142:0/1535860425 wait complete. 2024-03-25T19:34:44.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.362+0000 7ff8437f4640 1 librados: shutdown 2024-03-25T19:34:44.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 90d5c46a-9060-4cbc-82d5-7bed26691884 4 2024-03-25T19:34:44.375 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:44.376+0000 7f21f5c43040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:34:44.399 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:44.401+0000 7fae4fc30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.472+0000 7f9bca282640 10 monclient: get_monmap_and_config 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.472+0000 7f9bca282640 10 monclient: build_initial_monmap 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.472+0000 7f9bca282640 10 monclient: monmap: 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:44.472370+0000 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:44.472370+0000 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:44.471 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:44.472 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:44.472 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.472+0000 7f9bca282640 1 Processor -- start 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.472+0000 7f9bca282640 1 -- start start 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.472+0000 7f9bca282640 10 monclient: init 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc40b4700 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc4142a20 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc40a96b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc40a9d60 con 0x7f9bc4142a20 2024-03-25T19:34:44.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc413ff10 con 0x7f9bc40a96b0 2024-03-25T19:34:44.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc40aa1f0 con 0x7f9bc40b4700 2024-03-25T19:34:44.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:44.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.473+0000 7f9bca282640 10 monclient(hunting): authenticate will time out at 5604.780776s 2024-03-25T19:34:44.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.474+0000 7f9bc881f640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9bc40a96b0 0x7f9bc413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47718/0 (socket says 172.21.15.142:47718) 2024-03-25T19:34:44.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.474+0000 7f9bc881f640 1 -- 172.21.15.142:0/4038761337 learned_addr learned my addr 172.21.15.142:0/4038761337 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:44.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.476+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473065367 0 0) 0x7f9bc413ff10 con 0x7f9bc40a96b0 2024-03-25T19:34:44.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.476+0000 7f9bc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.476+0000 7f9bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.476+0000 7f9bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.476+0000 7f9bc2ffd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:34:44.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.476+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb0003520 con 0x7f9bc40a96b0 2024-03-25T19:34:44.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900892646 0 0) 0x7f9bc40aa1f0 con 0x7f9bc40b4700 2024-03-25T19:34:44.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc413ff10 con 0x7f9bc40b4700 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430800688 0 0) 0x7f9bc40a9d60 con 0x7f9bc4142a20 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:34:44.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.477+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc40aa1f0 con 0x7f9bc4142a20 2024-03-25T19:34:44.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1071234848 0 0) 0x7f9bb0003520 con 0x7f9bc40a96b0 2024-03-25T19:34:44.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 >> v1:172.21.15.47:6790/0 conn(0x7f9bc40b4700 legacy=0x7f9bc4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 >> v1:172.21.15.47:6789/0 conn(0x7f9bc4142a20 legacy=0x7f9bc413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40b4ae0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bca282640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9bbc001a10 con 0x7f9bc40a96b0 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bca282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bc2ffd640 10 monclient: dump: 2024-03-25T19:34:44.479 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.479+0000 7f9bca282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bbc001b70 con 0x7f9bc40a96b0 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bc2ffd640 1 -- 172.21.15.142:0/4038761337 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9bbc001f80 con 0x7f9bc40a96b0 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bca282640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bca282640 4 set_mon_vals no callback set 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.481+0000 7f9bc2ffd640 10 monclient: dump: 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 10 monclient: shutdown 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 1 -- 172.21.15.142:0/4038761337 >> v1:172.21.15.142:6789/0 conn(0x7f9bc40a96b0 legacy=0x7f9bc413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 1 -- 172.21.15.142:0/4038761337 shutdown_connections 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 1 -- 172.21.15.142:0/4038761337 >> 172.21.15.142:0/4038761337 conn(0x7f9bc40668e0 msgr2=0x7f9bc4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 1 -- 172.21.15.142:0/4038761337 shutdown_connections 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 1 -- 172.21.15.142:0/4038761337 wait complete. 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 10 monclient: build_initial_monmap 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.482+0000 7f9bca282640 10 monclient: monmap: 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:44.485 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 librados: starting msgr at 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 librados: starting objecter 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 Processor -- start 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 -- start start 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 librados: setting wanted keys 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 librados: calling monclient init 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 10 monclient: init 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc40a96b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc40b4700 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:44.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc4142a20 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc413be20 con 0x7f9bc40b4700 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc40abb60 con 0x7f9bc40a96b0 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.483+0000 7f9bca282640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc41d7490 con 0x7f9bc4142a20 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bca282640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bca282640 10 monclient(hunting): authenticate will time out at 5604.791177s 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bc3fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9bc40a96b0 0x7f9bc413b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47724/0 (socket says 172.21.15.142:47724) 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bc3fff640 1 -- 172.21.15.142:0/2626464879 learned_addr learned my addr 172.21.15.142:0/2626464879 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bc0ff9640 10 client.?.objecter ms_handle_connect 0x7f9bc40a96b0 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bc0ff9640 10 client.?.objecter ms_handle_connect 0x7f9bc4142a20 2024-03-25T19:34:44.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.484+0000 7f9bc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.485+0000 7f9bc0ff9640 10 client.?.objecter ms_handle_connect 0x7f9bc40b4700 2024-03-25T19:34:44.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.485+0000 7f9bc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.486+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667617931 0 0) 0x7f9bc40abb60 con 0x7f9bc40a96b0 2024-03-25T19:34:44.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.486+0000 7f9bc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.486+0000 7f9bc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.486+0000 7f9bc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.486+0000 7f9bc0ff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.486+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba4004eb0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920475367 0 0) 0x7f9bc41d7490 con 0x7f9bc4142a20 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc40abb60 con 0x7f9bc4142a20 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723519695 0 0) 0x7f9bc413be20 con 0x7f9bc40b4700 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:34:44.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.487+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc41d7490 con 0x7f9bc40b4700 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214133286 0 0) 0x7f9ba4004eb0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bc413be20 con 0x7f9bc40a96b0 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9bb4001a30 con 0x7f9bc40a96b0 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:44.490 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.491+0000 7f9bc0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.493+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 807650917 0 0) 0x7f9bc40abb60 con 0x7f9bc4142a20 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.493+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ba4004eb0 con 0x7f9bc4142a20 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.493+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9bbc001e80 con 0x7f9bc4142a20 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.493+0000 7f9bc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.493+0000 7f9bc0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.493+0000 7f9bc0ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070915488 0 0) 0x7f9bc41d7490 con 0x7f9bc40b4700 2024-03-25T19:34:44.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bc40abb60 con 0x7f9bc40b4700 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9bac001830 con 0x7f9bc40b4700 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.495 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2116349776 0 0) 0x7f9bc413be20 con 0x7f9bc40a96b0 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.494+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 >> v1:172.21.15.47:6790/0 conn(0x7f9bc4142a20 legacy=0x7f9bc41d4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 >> v1:172.21.15.47:6789/0 conn(0x7f9bc40b4700 legacy=0x7f9bc41d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 1 monclient: found mon.b 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc41d3fd0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 monclient: _renew_subs 2024-03-25T19:34:44.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc41d80c0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 monclient: _renew_subs 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc413c230 con 0x7f9bc40a96b0 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 1 librados: init done 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.495+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc41d7620 con 0x7f9bc40a96b0 2024-03-25T19:34:44.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.496+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bb4001c70 con 0x7f9bc40a96b0 2024-03-25T19:34:44.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.496+0000 7f9bc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:44.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.496+0000 7f9bc27fc640 4 set_mon_vals no callback set 2024-03-25T19:34:44.542 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:44.543+0000 7f21f5c43040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:34:44.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9bb4003a60 con 0x7f9bc40a96b0 2024-03-25T19:34:44.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:44.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 10 monclient: dump: 2024-03-25T19:34:44.594 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9bb4003fd0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2354589517 0 0) 0x7f9bb4006800 con 0x7f9bc40a96b0 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 10 client.4138.objecter ms_dispatch 0x7f9bc4000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:34:44.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:34:44.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T19:34:44.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9bb4006c00 con 0x7f9bc40a96b0 2024-03-25T19:34:44.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.595+0000 7f9bc0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:44.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.667+0000 7f9bca282640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "90d5c46a-9060-4cbc-82d5-7bed26691884", "id": 4}] 2024-03-25T19:34:44.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.667+0000 7f9bca282640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "90d5c46a-9060-4cbc-82d5-7bed26691884", "id": 4}] 2024-03-25T19:34:44.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.667+0000 7f9bca282640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.667+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd new", "uuid": "90d5c46a-9060-4cbc-82d5-7bed26691884", "id": 4} v 0) v1 -- 0x7f9bc404f730 con 0x7f9bc40a96b0 2024-03-25T19:34:44.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bc0ff9640 1 -- 172.21.15.142:0/2626464879 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90d5c46a-9060-4cbc-82d5-7bed26691884", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 643259969 0 1697012583) 0x7f9bb40062f0 con 0x7f9bc40a96b0 2024-03-25T19:34:44.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "90d5c46a-9060-4cbc-82d5-7bed26691884", "id": 4}] 2024-03-25T19:34:44.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bc0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:44.728 INFO:teuthology.orchestra.run.smithi142.stdout:4 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bca282640 10 librados: watch_flush enter 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bca282640 10 librados: watch_flush exit 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bca282640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.729+0000 7f9bca282640 10 client.4138.objecter successfully canceled tick 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 10 monclient: shutdown 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:44.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 >> v1:172.21.15.142:6789/0 conn(0x7f9bc40a96b0 legacy=0x7f9bc413b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 shutdown_connections 2024-03-25T19:34:44.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 >> 172.21.15.142:0/2626464879 conn(0x7f9bc40668e0 msgr2=0x7f9bc413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:44.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 shutdown_connections 2024-03-25T19:34:44.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 1 -- 172.21.15.142:0/2626464879 wait complete. 2024-03-25T19:34:44.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.730+0000 7f9bca282640 1 librados: shutdown 2024-03-25T19:34:44.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new c80800cd-ba3d-4a90-96fa-f3ba622e9836 5 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.882+0000 7fbf95cc1640 10 monclient: get_monmap_and_config 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.882+0000 7fbf95cc1640 10 monclient: build_initial_monmap 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.882+0000 7fbf95cc1640 10 monclient: monmap: 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:44.882732+0000 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:44.882732+0000 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:44.881 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:44.882 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:44.882 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 1 Processor -- start 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 1 -- start start 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient: init 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient(hunting): picked mon.noname-a con 0x7fbf900a9920 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900b45a0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient(hunting): picked mon.noname-b con 0x7fbf90146450 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9006b100 con 0x7fbf900a9920 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b2940 con 0x7fbf900b45a0 2024-03-25T19:34:44.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9013fec0 con 0x7fbf90146450 2024-03-25T19:34:44.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:44.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.883+0000 7fbf95cc1640 10 monclient(hunting): authenticate will time out at 5605.191084s 2024-03-25T19:34:44.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.884+0000 7fbf8effd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbf900b45a0 0x7fbf90145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47734/0 (socket says 172.21.15.142:47734) 2024-03-25T19:34:44.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.884+0000 7fbf8effd640 1 -- 172.21.15.142:0/3757473138 learned_addr learned my addr 172.21.15.142:0/3757473138 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.886+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956222687 0 0) 0x7fbf900b2940 con 0x7fbf900b45a0 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.886+0000 7fbf8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.886+0000 7fbf8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.886+0000 7fbf8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.886+0000 7fbf8e7fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.886+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf78003520 con 0x7fbf900b45a0 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531892669 0 0) 0x7fbf9013fec0 con 0x7fbf90146450 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:34:44.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900b2940 con 0x7fbf90146450 2024-03-25T19:34:44.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028823122 0 0) 0x7fbf9006b100 con 0x7fbf900a9920 2024-03-25T19:34:44.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:34:44.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.887+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9013fec0 con 0x7fbf900a9920 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2261542352 0 0) 0x7fbf78003520 con 0x7fbf900b45a0 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 >> v1:172.21.15.47:6790/0 conn(0x7fbf90146450 legacy=0x7fbf9014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 >> v1:172.21.15.47:6789/0 conn(0x7fbf900a9920 legacy=0x7fbf9013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf90140050 con 0x7fbf900b45a0 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf95cc1640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.889+0000 7fbf95cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.890+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbf840014b0 con 0x7fbf900b45a0 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.890+0000 7fbf8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.890+0000 7fbf8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.890+0000 7fbf8e7fc640 10 monclient: dump: 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.891 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.891 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.890+0000 7fbf95cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf84001700 con 0x7fbf900b45a0 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf8e7fc640 1 -- 172.21.15.142:0/3757473138 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbf84001a90 con 0x7fbf900b45a0 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf95cc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf95cc1640 4 set_mon_vals no callback set 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.891+0000 7fbf8e7fc640 10 monclient: dump: 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 10 monclient: shutdown 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 1 -- 172.21.15.142:0/3757473138 >> v1:172.21.15.142:6789/0 conn(0x7fbf900b45a0 legacy=0x7fbf90145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 1 -- 172.21.15.142:0/3757473138 shutdown_connections 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 1 -- 172.21.15.142:0/3757473138 >> 172.21.15.142:0/3757473138 conn(0x7fbf900b7f40 msgr2=0x7fbf900b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 1 -- 172.21.15.142:0/3757473138 shutdown_connections 2024-03-25T19:34:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.892+0000 7fbf95cc1640 1 -- 172.21.15.142:0/3757473138 wait complete. 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 10 monclient: build_initial_monmap 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 10 monclient: monmap: 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 1 librados: starting msgr at 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 1 librados: starting objecter 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 1 Processor -- start 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 1 -- start start 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 1 librados: setting wanted keys 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 1 librados: calling monclient init 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.893+0000 7fbf95cc1640 10 monclient: init 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900a9920 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient(hunting): picked mon.noname-a con 0x7fbf900b45a0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient(hunting): picked mon.noname-b con 0x7fbf90146450 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf901d7120 con 0x7fbf900b45a0 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf90141340 con 0x7fbf90146450 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b1730 con 0x7fbf900a9920 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf95cc1640 10 monclient(hunting): authenticate will time out at 5605.201573s 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf8ffff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbf90146450 0x7fbf90136fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47748/0 (socket says 172.21.15.142:47748) 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.894+0000 7fbf8ffff640 1 -- 172.21.15.142:0/2266134833 learned_addr learned my addr 172.21.15.142:0/2266134833 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.895+0000 7fbf77fff640 10 client.?.objecter ms_handle_connect 0x7fbf90146450 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.895+0000 7fbf77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.895+0000 7fbf77fff640 10 client.?.objecter ms_handle_connect 0x7fbf900a9920 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.895+0000 7fbf77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.895+0000 7fbf77fff640 10 client.?.objecter ms_handle_connect 0x7fbf900b45a0 2024-03-25T19:34:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.895+0000 7fbf77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.896+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809970890 0 0) 0x7fbf90141340 con 0x7fbf90146450 2024-03-25T19:34:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.896+0000 7fbf77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.896+0000 7fbf77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.896+0000 7fbf77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.896+0000 7fbf77fff640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.896+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf7c004eb0 con 0x7fbf90146450 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.897+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99326639 0 0) 0x7fbf900b1730 con 0x7fbf900a9920 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.897+0000 7fbf77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.897+0000 7fbf77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.897+0000 7fbf77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.897+0000 7fbf77fff640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.897+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf90141340 con 0x7fbf900a9920 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.898+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163818715 0 0) 0x7fbf901d7120 con 0x7fbf900b45a0 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.898+0000 7fbf77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.898+0000 7fbf77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.898+0000 7fbf77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.898+0000 7fbf77fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:34:44.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.898+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900b1730 con 0x7fbf900b45a0 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936552634 0 0) 0x7fbf7c004eb0 con 0x7fbf90146450 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf901d7120 con 0x7fbf90146450 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbf88001c00 con 0x7fbf90146450 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 10 monclient(hunting): dump: 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.902 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.902 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.902 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.902 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.902 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.901+0000 7fbf77fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:44.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.903+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884775492 0 0) 0x7fbf90141340 con 0x7fbf900a9920 2024-03-25T19:34:44.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.903+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf7c004eb0 con 0x7fbf900a9920 2024-03-25T19:34:44.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.903+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbf80001870 con 0x7fbf900a9920 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.903+0000 7fbf77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.903+0000 7fbf77fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.903+0000 7fbf77fff640 10 monclient(hunting): dump: 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2349606072 0 0) 0x7fbf900b1730 con 0x7fbf900b45a0 2024-03-25T19:34:44.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf90141340 con 0x7fbf900b45a0 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbf84001020 con 0x7fbf900b45a0 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient(hunting): dump: 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:44.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 982499262 0 0) 0x7fbf901d7120 con 0x7fbf90146450 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 >> v1:172.21.15.47:6790/0 conn(0x7fbf900a9920 legacy=0x7fbf901437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 >> v1:172.21.15.47:6789/0 conn(0x7fbf900b45a0 legacy=0x7fbf901d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 monclient: found mon.b 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf90143ec0 con 0x7fbf90146450 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 10 monclient: _renew_subs 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf90144880 con 0x7fbf90146450 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:44.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 10 monclient: _renew_subs 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf901d7f30 con 0x7fbf90146450 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.904+0000 7fbf95cc1640 1 librados: init done 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.905+0000 7fbf95cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.905+0000 7fbf95cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.905+0000 7fbf95cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:44.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:44.905+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf901d72b0 con 0x7fbf90146450 2024-03-25T19:34:45.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf880020d0 con 0x7fbf90146450 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbf88003b30 con 0x7fbf90146450 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 monclient: dump: 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.002 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbf880040e0 con 0x7fbf90146450 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf8dffb640 4 set_mon_vals no callback set 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3389524067 0 0) 0x7fbf88006a50 con 0x7fbf90146450 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 client.4144.objecter ms_dispatch 0x7fbf90000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbf88018de0 con 0x7fbf90146450 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.002+0000 7fbf77fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:45.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.075+0000 7fbf95cc1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c80800cd-ba3d-4a90-96fa-f3ba622e9836", "id": 5}] 2024-03-25T19:34:45.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.075+0000 7fbf95cc1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c80800cd-ba3d-4a90-96fa-f3ba622e9836", "id": 5}] 2024-03-25T19:34:45.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.075+0000 7fbf95cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.075+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c80800cd-ba3d-4a90-96fa-f3ba622e9836", "id": 5} v 0) v1 -- 0x7fbf9004f730 con 0x7fbf90146450 2024-03-25T19:34:45.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.136+0000 7fbf77fff640 1 -- 172.21.15.142:0/2266134833 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c80800cd-ba3d-4a90-96fa-f3ba622e9836", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1624219073 0 1988412176) 0x7fbf88004460 con 0x7fbf90146450 2024-03-25T19:34:45.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.136+0000 7fbf77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c80800cd-ba3d-4a90-96fa-f3ba622e9836", "id": 5}] 2024-03-25T19:34:45.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.136+0000 7fbf77fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:45.135 INFO:teuthology.orchestra.run.smithi142.stdout:5 2024-03-25T19:34:45.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 10 librados: watch_flush enter 2024-03-25T19:34:45.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 10 librados: watch_flush exit 2024-03-25T19:34:45.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:34:45.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:34:45.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 10 monclient: shutdown 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 >> v1:172.21.15.142:6789/0 conn(0x7fbf90146450 legacy=0x7fbf90136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 shutdown_connections 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 >> 172.21.15.142:0/2266134833 conn(0x7fbf900b7f40 msgr2=0x7fbf90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 shutdown_connections 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 1 -- 172.21.15.142:0/2266134833 wait complete. 2024-03-25T19:34:45.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.137+0000 7fbf95cc1640 1 librados: shutdown 2024-03-25T19:34:45.147 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new ee8e69c9-9b53-4daa-a98d-8b481464375a 6 2024-03-25T19:34:45.232 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:45.234+0000 7fae4fc30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:34:45.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.248+0000 7fcbe0a1e640 10 monclient: get_monmap_and_config 2024-03-25T19:34:45.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.249+0000 7fcbe0a1e640 10 monclient: build_initial_monmap 2024-03-25T19:34:45.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.249+0000 7fcbe0a1e640 10 monclient: monmap: 2024-03-25T19:34:45.247 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:45.249234+0000 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:45.249234+0000 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:45.248 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.249+0000 7fcbe0a1e640 1 Processor -- start 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.249+0000 7fcbe0a1e640 1 -- start start 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.249+0000 7fcbe0a1e640 10 monclient: init 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fcbdc0a9920 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc0b45a0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fcbdc146450 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc06b100 con 0x7fcbdc146450 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0b2940 con 0x7fcbdc0a9920 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc13fec0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbe0a1e640 10 monclient(hunting): authenticate will time out at 5605.557589s 2024-03-25T19:34:45.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbda59c640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcbdc0a9920 0x7fcbdc13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47762/0 (socket says 172.21.15.142:47762) 2024-03-25T19:34:45.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.250+0000 7fcbda59c640 1 -- 172.21.15.142:0/4055034848 learned_addr learned my addr 172.21.15.142:0/4055034848 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:45.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940210455 0 0) 0x7fcbdc0b2940 con 0x7fcbdc0a9920 2024-03-25T19:34:45.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc4003520 con 0x7fcbdc0a9920 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682731554 0 0) 0x7fcbdc06b100 con 0x7fcbdc146450 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.253+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc0b2940 con 0x7fcbdc146450 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.254+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703556630 0 0) 0x7fcbdc13fec0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.254+0000 7fcbd959a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.254+0000 7fcbd959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.254+0000 7fcbd959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.254+0000 7fcbd959a640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:34:45.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.254+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc06b100 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344679510 0 0) 0x7fcbc4003520 con 0x7fcbdc0a9920 2024-03-25T19:34:45.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 >> v1:172.21.15.47:6790/0 conn(0x7fcbdc0b45a0 legacy=0x7fcbdc145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 >> v1:172.21.15.47:6789/0 conn(0x7fcbdc146450 legacy=0x7fcbdc14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: _finish_hunting 0 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 1 monclient: found mon.noname-c 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc140050 con 0x7fcbdc0a9920 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: _finish_auth 0 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: _check_auth_tickets 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fcbcc001690 con 0x7fcbdc0a9920 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbe0a1e640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbe0a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:45.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbd959a640 10 monclient: dump: 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.256+0000 7fcbe0a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbcc0017f0 con 0x7fcbdc0a9920 2024-03-25T19:34:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbd959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbd959a640 1 -- 172.21.15.142:0/4055034848 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fcbcc001a60 con 0x7fcbdc0a9920 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbe0a1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbe0a1e640 4 set_mon_vals no callback set 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbd959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbd959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbd959a640 10 monclient: dump: 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbe0a1e640 10 monclient: shutdown 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbe0a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.258+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/4055034848 >> v1:172.21.15.142:6789/0 conn(0x7fcbdc0a9920 legacy=0x7fcbdc13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.259+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/4055034848 shutdown_connections 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.259+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/4055034848 >> 172.21.15.142:0/4055034848 conn(0x7fcbdc0b7f40 msgr2=0x7fcbdc0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.259+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/4055034848 shutdown_connections 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.259+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/4055034848 wait complete. 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.259+0000 7fcbe0a1e640 10 monclient: build_initial_monmap 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.259+0000 7fcbe0a1e640 10 monclient: monmap: 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:45.262 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 librados: starting msgr at 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 librados: starting objecter 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 Processor -- start 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 -- start start 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 librados: setting wanted keys 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 librados: calling monclient init 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient: init 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:45.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fcbdc0a9920 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fcbdc0b45a0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fcbdc146450 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0ac090 con 0x7fcbdc0a9920 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0ac370 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0ac650 con 0x7fcbdc146450 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.260+0000 7fcbe0a1e640 10 monclient(hunting): authenticate will time out at 5605.568087s 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbd9d9b640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcbdc0b45a0 0x7fcbdc0b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47764/0 (socket says 172.21.15.142:47764) 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbd9d9b640 1 -- 172.21.15.142:0/1261704569 learned_addr learned my addr 172.21.15.142:0/1261704569 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbcaffd640 10 client.?.objecter ms_handle_connect 0x7fcbdc0b45a0 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbcaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbcaffd640 10 client.?.objecter ms_handle_connect 0x7fcbdc146450 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbcaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbcaffd640 10 client.?.objecter ms_handle_connect 0x7fcbdc0a9920 2024-03-25T19:34:45.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.261+0000 7fcbcaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:45.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.263+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284476820 0 0) 0x7fcbdc0ac370 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.263+0000 7fcbcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.263+0000 7fcbcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.263+0000 7fcbcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.263+0000 7fcbcaffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.263+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbbc004eb0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131990714 0 0) 0x7fcbdc0ac090 con 0x7fcbdc0a9920 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc0ac370 con 0x7fcbdc0a9920 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319866239 0 0) 0x7fcbdc0ac650 con 0x7fcbdc146450 2024-03-25T19:34:45.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:34:45.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.264+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc0ac090 con 0x7fcbdc146450 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484803416 0 0) 0x7fcbbc004eb0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbdc0ac650 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fcbd0001a10 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 10 monclient(hunting): dump: 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.268 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.268+0000 7fcbcaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.269+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1085479085 0 0) 0x7fcbdc0ac370 con 0x7fcbdc0a9920 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.269+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbbc004eb0 con 0x7fcbdc0a9920 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.269+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fcbcc001430 con 0x7fcbdc0a9920 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.269+0000 7fcbcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.269+0000 7fcbcaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.269+0000 7fcbcaffd640 10 monclient(hunting): dump: 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.269 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.270 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.270 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.270 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.270 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.270 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 207005095 0 0) 0x7fcbdc0ac650 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 >> v1:172.21.15.47:6790/0 conn(0x7fcbdc146450 legacy=0x7fcbdc0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 >> v1:172.21.15.47:6789/0 conn(0x7fcbdc0a9920 legacy=0x7fcbdc1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 1 monclient: found mon.b 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc144090 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 10 monclient: _renew_subs 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc144340 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 10 monclient: _renew_subs 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc144ca0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.271+0000 7fcbe0a1e640 1 librados: init done 2024-03-25T19:34:45.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.272+0000 7fcbe0a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.272+0000 7fcbe0a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.272+0000 7fcbe0a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.272+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc1da810 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd0001c70 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:45.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fcbd00038c0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 monclient: dump: 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbd8d99640 4 set_mon_vals no callback set 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcbd0003e70 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2570390941 0 0) 0x7fcbd0006680 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 client.4150.objecter ms_dispatch 0x7fcbdc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:34:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:34:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:34:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:34:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcbd00069a0 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.370+0000 7fcbcaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:45.425 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:45.426+0000 7f21f5c43040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:34:45.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.449+0000 7fcbe0a1e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ee8e69c9-9b53-4daa-a98d-8b481464375a", "id": 6}] 2024-03-25T19:34:45.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.449+0000 7fcbe0a1e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ee8e69c9-9b53-4daa-a98d-8b481464375a", "id": 6}] 2024-03-25T19:34:45.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.449+0000 7fcbe0a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.449+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ee8e69c9-9b53-4daa-a98d-8b481464375a", "id": 6} v 0) v1 -- 0x7fcbdc04f730 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.513+0000 7fcbcaffd640 1 -- 172.21.15.142:0/1261704569 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee8e69c9-9b53-4daa-a98d-8b481464375a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2602217967 0 1113811849) 0x7fcbd0004420 con 0x7fcbdc0b45a0 2024-03-25T19:34:45.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.513+0000 7fcbcaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ee8e69c9-9b53-4daa-a98d-8b481464375a", "id": 6}] 2024-03-25T19:34:45.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.513+0000 7fcbcaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:45.512 INFO:teuthology.orchestra.run.smithi142.stdout:6 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 10 librados: watch_flush enter 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 10 librados: watch_flush exit 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 10 monclient: shutdown 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 >> v1:172.21.15.142:6789/0 conn(0x7fcbdc0b45a0 legacy=0x7fcbdc0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 shutdown_connections 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 >> 172.21.15.142:0/1261704569 conn(0x7fcbdc0b7f40 msgr2=0x7fcbdc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 shutdown_connections 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 1 -- 172.21.15.142:0/1261704569 wait complete. 2024-03-25T19:34:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.514+0000 7fcbe0a1e640 1 librados: shutdown 2024-03-25T19:34:45.520 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:45.521+0000 7fae4fc30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:34:45.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 451b8a1c-8ed2-41d1-9fff-16fe77956190 7 2024-03-25T19:34:45.575 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:45.576+0000 7fae4fc30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.624+0000 7fe0db11c640 10 monclient: get_monmap_and_config 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.624+0000 7fe0db11c640 10 monclient: build_initial_monmap 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.624+0000 7fe0db11c640 10 monclient: monmap: 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:34:45.624871+0000 2024-03-25T19:34:45.623 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:34:45.624871+0000 2024-03-25T19:34:45.624 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:45.624 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.624 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:45.624 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:45.624 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:45.624 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.625+0000 7fe0db11c640 1 Processor -- start 2024-03-25T19:34:45.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.625+0000 7fe0db11c640 1 -- start start 2024-03-25T19:34:45.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.625+0000 7fe0db11c640 10 monclient: init 2024-03-25T19:34:45.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.625+0000 7fe0db11c640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:45.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.625+0000 7fe0db11c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d40b4700 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d4142a20 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d40a96b0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40a9d60 con 0x7fe0d40b4700 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d413ff10 con 0x7fe0d4142a20 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40aa1f0 con 0x7fe0d40a96b0 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0db11c640 10 monclient(hunting): authenticate will time out at 5605.933252s 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0d3fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe0d4142a20 0x7fe0d413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47768/0 (socket says 172.21.15.142:47768) 2024-03-25T19:34:45.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.626+0000 7fe0d3fff640 1 -- 172.21.15.142:0/2992639766 learned_addr learned my addr 172.21.15.142:0/2992639766 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:45.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738453471 0 0) 0x7fe0d413ff10 con 0x7fe0d4142a20 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c4003520 con 0x7fe0d4142a20 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049649029 0 0) 0x7fe0d40a9d60 con 0x7fe0d40b4700 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:34:45.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d413ff10 con 0x7fe0d40b4700 2024-03-25T19:34:45.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381021511 0 0) 0x7fe0d40aa1f0 con 0x7fe0d40a96b0 2024-03-25T19:34:45.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:34:45.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.629+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d40a9d60 con 0x7fe0d40a96b0 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465079673 0 0) 0x7fe0d413ff10 con 0x7fe0d40b4700 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 >> v1:172.21.15.47:6790/0 conn(0x7fe0d40a96b0 legacy=0x7fe0d413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 >> v1:172.21.15.142:6789/0 conn(0x7fe0d4142a20 legacy=0x7fe0d413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 1 monclient: found mon.noname-a 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40b4ae0 con 0x7fe0d40b4700 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:45.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0db11c640 5 monclient: authenticate success, global_id 4158 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0db11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe0c8001670 con 0x7fe0d40b4700 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0d37fe640 10 monclient: dump: 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.632 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.631+0000 7fe0db11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:45.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c8001850 con 0x7fe0d40b4700 2024-03-25T19:34:45.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:45.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0db11c640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0db11c640 4 set_mon_vals no callback set 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0d37fe640 1 -- 172.21.15.142:0/2992639766 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe0c8001c60 con 0x7fe0d40b4700 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0d37fe640 10 monclient: dump: 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.635 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0db11c640 10 monclient: shutdown 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0db11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.634+0000 7fe0db11c640 1 -- 172.21.15.142:0/2992639766 >> v1:172.21.15.47:6789/0 conn(0x7fe0d40b4700 legacy=0x7fe0d4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 1 -- 172.21.15.142:0/2992639766 shutdown_connections 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 1 -- 172.21.15.142:0/2992639766 >> 172.21.15.142:0/2992639766 conn(0x7fe0d40668e0 msgr2=0x7fe0d4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 1 -- 172.21.15.142:0/2992639766 shutdown_connections 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 1 -- 172.21.15.142:0/2992639766 wait complete. 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 10 monclient: build_initial_monmap 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 10 monclient: monmap: 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-03-25T19:34:45.636 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 1 librados: starting msgr at 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.635+0000 7fe0db11c640 1 librados: starting objecter 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 Processor -- start 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 -- start start 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 librados: setting wanted keys 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 librados: calling monclient init 2024-03-25T19:34:45.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient: init 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d40a96b0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d40b4700 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d4142a20 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4133e10 con 0x7fe0d4142a20 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d413c150 con 0x7fe0d40b4700 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40ab8f0 con 0x7fe0d40a96b0 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.636+0000 7fe0db11c640 10 monclient(hunting): authenticate will time out at 5605.944004s 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.637+0000 7fe0d8eb8640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe0d40a96b0 0x7fe0d40ac2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.142:44542/0 (socket says 172.21.15.142:44542) 2024-03-25T19:34:45.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.637+0000 7fe0d8eb8640 1 -- 172.21.15.142:0/2723050093 learned_addr learned my addr 172.21.15.142:0/2723050093 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.637+0000 7fe0d17fa640 10 client.?.objecter ms_handle_connect 0x7fe0d40a96b0 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.637+0000 7fe0d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.638+0000 7fe0d17fa640 10 client.?.objecter ms_handle_connect 0x7fe0d4142a20 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.638+0000 7fe0d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.638+0000 7fe0d17fa640 10 client.?.objecter ms_handle_connect 0x7fe0d40b4700 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.638+0000 7fe0d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026881101 0 0) 0x7fe0d40a9d60 con 0x7fe0d4142a20 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8004eb0 con 0x7fe0d4142a20 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171596010 0 0) 0x7fe0d40ab8f0 con 0x7fe0d40a96b0 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d40a9d60 con 0x7fe0d40a96b0 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430396191 0 0) 0x7fe0d413c150 con 0x7fe0d40b4700 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:34:45.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.640+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d40ab8f0 con 0x7fe0d40b4700 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441179502 0 0) 0x7fe0d40ab8f0 con 0x7fe0d40b4700 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d413c150 con 0x7fe0d40b4700 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe0c0001710 con 0x7fe0d40b4700 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 10 monclient(hunting): dump: 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.644 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.645+0000 7fe0d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754858558 0 0) 0x7fe0b8004eb0 con 0x7fe0d4142a20 2024-03-25T19:34:45.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d40ab8f0 con 0x7fe0d4142a20 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe0cc001ad0 con 0x7fe0d4142a20 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 10 monclient(hunting): dump: 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.646 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.647 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191699136 0 0) 0x7fe0d40a9d60 con 0x7fe0d40a96b0 2024-03-25T19:34:45.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0b8004eb0 con 0x7fe0d40a96b0 2024-03-25T19:34:45.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe0c8001c20 con 0x7fe0d40a96b0 2024-03-25T19:34:45.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.647+0000 7fe0d17fa640 10 monclient(hunting): dump: 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4014082210 0 0) 0x7fe0d413c150 con 0x7fe0d40b4700 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 >> v1:172.21.15.47:6790/0 conn(0x7fe0d40a96b0 legacy=0x7fe0d40ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 >> v1:172.21.15.47:6789/0 conn(0x7fe0d4142a20 legacy=0x7fe0d41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 1 monclient: found mon.b 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40abb60 con 0x7fe0d40b4700 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0db11c640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0db11c640 10 monclient: _renew_subs 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.648+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d40abe40 con 0x7fe0d40b4700 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:45.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 10 monclient: _renew_subs 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d413af20 con 0x7fe0d40b4700 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 1 librados: init done 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.649+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d404f730 con 0x7fe0d40b4700 2024-03-25T19:34:45.695 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:45.697+0000 7fae4fc30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:34:45.722 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:45.723+0000 7f21f5c43040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:34:45.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c0001c20 con 0x7fe0d40b4700 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe0c0004980 con 0x7fe0d40b4700 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 monclient: dump: 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 3 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:45.746 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe0c0004e70 con 0x7fe0d40b4700 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d2ffd640 4 set_mon_vals no callback set 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1367093898 0 0) 0x7fe0c0006670 con 0x7fe0d40b4700 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 client.4156.objecter ms_dispatch 0x7fe0d4000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe0c0006ad0 con 0x7fe0d40b4700 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:45.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.746+0000 7fe0d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:45.775 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:45.776+0000 7f21f5c43040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:34:45.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.817+0000 7fe0db11c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "451b8a1c-8ed2-41d1-9fff-16fe77956190", "id": 7}] 2024-03-25T19:34:45.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.817+0000 7fe0db11c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "451b8a1c-8ed2-41d1-9fff-16fe77956190", "id": 7}] 2024-03-25T19:34:45.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.817+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:45.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:45.817+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd new", "uuid": "451b8a1c-8ed2-41d1-9fff-16fe77956190", "id": 7} v 0) v1 -- 0x7fe0d40b7530 con 0x7fe0d40b4700 2024-03-25T19:34:45.915 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-25T19:34:45.916+0000 7f21f5c43040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:34:46.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.086+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4229068595 0 0) 0x7fe0c0005f90 con 0x7fe0d40b4700 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.416+0000 7fe0d17fa640 1 -- 172.21.15.142:0/2723050093 <== mon.1 v1:172.21.15.142:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "451b8a1c-8ed2-41d1-9fff-16fe77956190", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 785711062 0 1371665406) 0x7fe0c0015880 con 0x7fe0d40b4700 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.416+0000 7fe0d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "451b8a1c-8ed2-41d1-9fff-16fe77956190", "id": 7}] 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.416+0000 7fe0d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi142.stdout:7 2024-03-25T19:34:46.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 10 librados: watch_flush enter 2024-03-25T19:34:46.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 10 librados: watch_flush exit 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 10 monclient: shutdown 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 >> v1:172.21.15.142:6789/0 conn(0x7fe0d40b4700 legacy=0x7fe0d4139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 shutdown_connections 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 >> 172.21.15.142:0/2723050093 conn(0x7fe0d40668e0 msgr2=0x7fe0d40b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 shutdown_connections 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 1 -- 172.21.15.142:0/2723050093 wait complete. 2024-03-25T19:34:46.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-25T19:34:46.417+0000 7fe0db11c640 1 librados: shutdown 2024-03-25T19:34:46.427 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.427 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:34:46.427 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:34:46.429 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:34:46.430 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.430 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:34:46.430 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:34:46.432 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:34:46.432 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.433 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:34:46.433 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:34:46.435 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:34:46.435 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.435 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:34:46.435 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:34:46.437 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:34:46.437 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.438 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:34:46.438 DEBUG:teuthology.orchestra.run.smithi142:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:34:46.440 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:34:46.440 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.440 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:34:46.440 DEBUG:teuthology.orchestra.run.smithi142:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:34:46.442 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:34:46.442 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.442 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:34:46.442 DEBUG:teuthology.orchestra.run.smithi142:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:34:46.444 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:34:46.444 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:34:46.445 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:34:46.445 DEBUG:teuthology.orchestra.run.smithi142:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:34:46.446 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:34:46.447 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:46.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.593+0000 7fbec2bc8640 10 monclient: get_monmap_and_config 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.593+0000 7fbec2bc8640 10 monclient: build_initial_monmap 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.593+0000 7fbec2bc8640 10 monclient: monmap: 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:34:46.593948+0000 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:34:46.593948+0000 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:46.593 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.594+0000 7fbec2bc8640 1 Processor -- start 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.594+0000 7fbec2bc8640 1 -- start start 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.594+0000 7fbec2bc8640 10 monclient: init 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient(hunting): picked mon.noname-c con 0x7fbebc0b4700 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient(hunting): picked mon.noname-a con 0x7fbebc142a20 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:46.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient(hunting): picked mon.noname-b con 0x7fbebc0a96b0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0a9d60 con 0x7fbebc142a20 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc13ff10 con 0x7fbebc0b4700 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0aa1f0 con 0x7fbebc0a96b0 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.595+0000 7fbec2bc8640 10 monclient(hunting): authenticate will time out at 5601.881218s 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.596+0000 7fbebbfff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbebc142a20 0x7fbebc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60246/0 (socket says 172.21.15.47:60246) 2024-03-25T19:34:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.596+0000 7fbebbfff640 1 -- 172.21.15.47:0/2695378379 learned_addr learned my addr 172.21.15.47:0/2695378379 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.657+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073429382 0 0) 0x7fbebc0aa1f0 con 0x7fbebc0a96b0 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.657+0000 7fbebb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.657+0000 7fbebb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.657+0000 7fbebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.657+0000 7fbebb7fe640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.657+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeac003520 con 0x7fbebc0a96b0 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.658+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498855368 0 0) 0x7fbebc13ff10 con 0x7fbebc0b4700 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.658+0000 7fbebb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.658+0000 7fbebb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.658+0000 7fbebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.658+0000 7fbebb7fe640 10 monclient(hunting): my global_id is 4168 2024-03-25T19:34:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.658+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc0aa1f0 con 0x7fbebc0b4700 2024-03-25T19:34:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 709026781 0 0) 0x7fbebc0aa1f0 con 0x7fbebc0b4700 2024-03-25T19:34:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 >> v1:172.21.15.47:6790/0 conn(0x7fbebc0a96b0 legacy=0x7fbebc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 >> v1:172.21.15.47:6789/0 conn(0x7fbebc142a20 legacy=0x7fbebc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 1 monclient: found mon.noname-c 2024-03-25T19:34:46.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0b4ae0 con 0x7fbebc0b4700 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbec2bc8640 5 monclient: authenticate success, global_id 4168 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbec2bc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbeb00015f0 con 0x7fbebc0b4700 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:46.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbebb7fe640 10 monclient: dump: 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.681+0000 7fbec2bc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbeb00017d0 con 0x7fbebc0b4700 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbebb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbec2bc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbec2bc8640 4 set_mon_vals no callback set 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbebb7fe640 1 -- 172.21.15.47:0/2695378379 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbeb00015f0 con 0x7fbebc0b4700 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbebb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbebb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.700+0000 7fbebb7fe640 10 monclient: dump: 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.701+0000 7fbec2bc8640 10 monclient: shutdown 2024-03-25T19:34:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.701+0000 7fbec2bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.701+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2695378379 >> v1:172.21.15.142:6789/0 conn(0x7fbebc0b4700 legacy=0x7fbebc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.701+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2695378379 shutdown_connections 2024-03-25T19:34:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.701+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2695378379 >> 172.21.15.47:0/2695378379 conn(0x7fbebc0668e0 msgr2=0x7fbebc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.701+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2695378379 shutdown_connections 2024-03-25T19:34:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.702+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2695378379 wait complete. 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.702+0000 7fbec2bc8640 10 monclient: build_initial_monmap 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.702+0000 7fbec2bc8640 10 monclient: monmap: 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 1 librados: starting msgr at 2024-03-25T19:34:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 1 librados: starting objecter 2024-03-25T19:34:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 1 Processor -- start 2024-03-25T19:34:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 1 -- start start 2024-03-25T19:34:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 1 librados: setting wanted keys 2024-03-25T19:34:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 1 librados: calling monclient init 2024-03-25T19:34:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.703+0000 7fbec2bc8640 10 monclient: init 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.704+0000 7fbec2bc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.704+0000 7fbec2bc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.704+0000 7fbec2bc8640 10 monclient(hunting): picked mon.noname-c con 0x7fbebc0a96b0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.704+0000 7fbec2bc8640 10 monclient(hunting): picked mon.noname-b con 0x7fbebc0b4700 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.704+0000 7fbec2bc8640 10 monclient(hunting): picked mon.noname-a con 0x7fbebc142a20 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.704+0000 7fbec2bc8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc136e50 con 0x7fbebc142a20 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.705+0000 7fbec2bc8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc138470 con 0x7fbebc0b4700 2024-03-25T19:34:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.705+0000 7fbec2bc8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc1372e0 con 0x7fbebc0a96b0 2024-03-25T19:34:46.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.705+0000 7fbec2bc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:46.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.705+0000 7fbec2bc8640 10 monclient(hunting): authenticate will time out at 5601.991153s 2024-03-25T19:34:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.710+0000 7fbebbfff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbebc0b4700 0x7fbebc1dc280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:52228/0 (socket says 172.21.15.47:52228) 2024-03-25T19:34:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.710+0000 7fbebbfff640 1 -- 172.21.15.47:0/2507705882 learned_addr learned my addr 172.21.15.47:0/2507705882 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.711+0000 7fbeb97fa640 10 client.?.objecter ms_handle_connect 0x7fbebc0b4700 2024-03-25T19:34:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.711+0000 7fbeb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.715+0000 7fbeb97fa640 10 client.?.objecter ms_handle_connect 0x7fbebc0a96b0 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.715+0000 7fbeb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.716+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843476449 0 0) 0x7fbebc138470 con 0x7fbebc0b4700 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.716+0000 7fbeb97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.716+0000 7fbeb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.716+0000 7fbeb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.716+0000 7fbeb97fa640 10 monclient(hunting): my global_id is 4174 2024-03-25T19:34:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.716+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0004eb0 con 0x7fbebc0b4700 2024-03-25T19:34:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.719+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720879829 0 0) 0x7fbebc1372e0 con 0x7fbebc0a96b0 2024-03-25T19:34:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.719+0000 7fbeb97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.719+0000 7fbeb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.719+0000 7fbeb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.719+0000 7fbeb97fa640 10 monclient(hunting): my global_id is 4169 2024-03-25T19:34:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.719+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc138470 con 0x7fbebc0a96b0 2024-03-25T19:34:46.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.726+0000 7fbeb97fa640 10 client.?.objecter ms_handle_connect 0x7fbebc142a20 2024-03-25T19:34:46.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.726+0000 7fbeb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.728+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191409589 0 0) 0x7fbea0004eb0 con 0x7fbebc0b4700 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.728+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbebc1372e0 con 0x7fbebc0b4700 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbea8001810 con 0x7fbebc0b4700 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): dump: 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:46.728 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359984682 0 0) 0x7fbebc138470 con 0x7fbebc0a96b0 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbea0004eb0 con 0x7fbebc0a96b0 2024-03-25T19:34:46.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbeb00015f0 con 0x7fbebc0a96b0 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.729+0000 7fbeb97fa640 10 monclient(hunting): dump: 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:46.730 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2246153950 0 0) 0x7fbebc1372e0 con 0x7fbebc0b4700 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 >> v1:172.21.15.47:6790/0 conn(0x7fbebc0a96b0 legacy=0x7fbebc1dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 >> v1:172.21.15.47:6789/0 conn(0x7fbebc142a20 legacy=0x7fbebc1e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 1 monclient: found mon.b 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc06e7a0 con 0x7fbebc0b4700 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbeb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbec2bc8640 5 monclient: authenticate success, global_id 4174 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbec2bc8640 10 monclient: _renew_subs 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbec2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.736+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc071bd0 con 0x7fbebc0b4700 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.737+0000 7fbec2bc8640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.737+0000 7fbec2bc8640 10 monclient: _renew_subs 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.737+0000 7fbec2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.737+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc0712a0 con 0x7fbebc0b4700 2024-03-25T19:34:46.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.737+0000 7fbec2bc8640 1 librados: init done 2024-03-25T19:34:46.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.738+0000 7fbec2bc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:46.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.738+0000 7fbec2bc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:46.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.738+0000 7fbec2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:46.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.738+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc137000 con 0x7fbebc0b4700 2024-03-25T19:34:46.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbea8001970 con 0x7fbebc0b4700 2024-03-25T19:34:46.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbeb97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbea80047e0 con 0x7fbebc0b4700 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbeb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbeb97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbeb97fa640 10 monclient: dump: 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.939+0000 7fbebaffd640 4 set_mon_vals no callback set 2024-03-25T19:34:46.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.940+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4229068595 0 0) 0x7fbea8004a00 con 0x7fbebc0b4700 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fbea8015490 con 0x7fbebc0b4700 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 10 client.4174.objecter ms_dispatch 0x7fbebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 3 client.4174.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 3 client.4174.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 20 client.4174.objecter dump_active .. 0 homeless 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbea8014750 con 0x7fbebc0b4700 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:46.944+0000 7fbeb97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:47.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.070+0000 7fbec2bc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:47.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.070+0000 7fbec2bc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:47.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.070+0000 7fbec2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:47.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.070+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbebc06e990 con 0x7fbebc0b4700 2024-03-25T19:34:47.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.142+0000 7fbeb97fa640 1 -- 172.21.15.47:0/2507705882 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fbea8014750 con 0x7fbebc0b4700 2024-03-25T19:34:47.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.142+0000 7fbeb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:47.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.142+0000 7fbeb97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:47.142 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-25T19:34:47.142 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:47.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 10 librados: watch_flush enter 2024-03-25T19:34:47.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 10 librados: watch_flush exit 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 20 client.4174.objecter shutdown clearing up homeless session... 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 10 client.4174.objecter successfully canceled tick 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 10 monclient: shutdown 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 >> v1:172.21.15.142:6789/0 conn(0x7fbebc0b4700 legacy=0x7fbebc1dc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 shutdown_connections 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 >> 172.21.15.47:0/2507705882 conn(0x7fbebc0668e0 msgr2=0x7fbebc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 shutdown_connections 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 1 -- 172.21.15.47:0/2507705882 wait complete. 2024-03-25T19:34:47.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-25T19:34:47.144+0000 7fbec2bc8640 1 librados: shutdown 2024-03-25T19:34:47.159 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:34:47.159 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:34:47.593 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:47.594+0000 7fadf01c3640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:34:47.765 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:34:47.766+0000 7fadf01c3640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:34:47.965 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:48.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.118+0000 7f22f4486640 10 monclient: get_monmap_and_config 2024-03-25T19:34:48.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.118+0000 7f22f4486640 10 monclient: build_initial_monmap 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.118+0000 7f22f4486640 10 monclient: monmap: 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:34:48.118895+0000 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:34:48.118895+0000 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:48.118 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.119+0000 7f22f4486640 1 Processor -- start 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.119+0000 7f22f4486640 1 -- start start 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.119+0000 7f22f4486640 10 monclient: init 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec142520 con 0x7f22ec13f710 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0b28a0 con 0x7f22ec1430c0 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0b2a30 con 0x7f22ec0a98f0 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:48.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.120+0000 7f22f4486640 10 monclient(hunting): authenticate will time out at 5603.406239s 2024-03-25T19:34:48.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.123+0000 7f22f1a21640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f22ec13f710 0x7f22ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60816/0 (socket says 172.21.15.47:60816) 2024-03-25T19:34:48.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.123+0000 7f22f1a21640 1 -- 172.21.15.47:0/3245901657 learned_addr learned my addr 172.21.15.47:0/3245901657 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:48.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.128+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433508067 0 0) 0x7f22ec142520 con 0x7f22ec13f710 2024-03-25T19:34:48.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.128+0000 7f22f1220640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.128+0000 7f22f1220640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.128+0000 7f22f1220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.128+0000 7f22f1220640 10 monclient(hunting): my global_id is 4182 2024-03-25T19:34:48.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.128+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d8003520 con 0x7f22ec13f710 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 330382513 0 0) 0x7f22d8003520 con 0x7f22ec13f710 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 >> v1:172.21.15.47:6790/0 conn(0x7f22ec0a98f0 legacy=0x7f22ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 >> v1:172.21.15.142:6789/0 conn(0x7f22ec1430c0 legacy=0x7f22ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: _finish_hunting 0 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 1 monclient: found mon.noname-a 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec0b44c0 con 0x7f22ec13f710 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: _finish_auth 0 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: _check_auth_tickets 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f22e8001690 con 0x7f22ec13f710 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f1220640 10 monclient: dump: 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.136 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f4486640 5 monclient: authenticate success, global_id 4182 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.134+0000 7f22f4486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:48.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e8001860 con 0x7f22ec13f710 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f1220640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f1220640 1 -- 172.21.15.47:0/3245901657 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f22e8001cb0 con 0x7f22ec13f710 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f1220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f1220640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f1220640 10 monclient: dump: 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:48.140 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f4486640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.137+0000 7f22f4486640 4 set_mon_vals no callback set 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.138+0000 7f22f4486640 10 monclient: shutdown 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.138+0000 7f22f4486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.138+0000 7f22f4486640 1 -- 172.21.15.47:0/3245901657 >> v1:172.21.15.47:6789/0 conn(0x7f22ec13f710 legacy=0x7f22ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.138+0000 7f22f4486640 1 -- 172.21.15.47:0/3245901657 shutdown_connections 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.138+0000 7f22f4486640 1 -- 172.21.15.47:0/3245901657 >> 172.21.15.47:0/3245901657 conn(0x7f22ec0b7ea0 msgr2=0x7f22ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.138+0000 7f22f4486640 1 -- 172.21.15.47:0/3245901657 shutdown_connections 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.139+0000 7f22f4486640 1 -- 172.21.15.47:0/3245901657 wait complete. 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.139+0000 7f22f4486640 10 monclient: build_initial_monmap 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.139+0000 7f22f4486640 10 monclient: monmap: 2024-03-25T19:34:48.141 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 1 librados: starting msgr at 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 1 librados: starting objecter 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 1 Processor -- start 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 1 -- start start 2024-03-25T19:34:48.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 1 librados: setting wanted keys 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 1 librados: calling monclient init 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.140+0000 7f22f4486640 10 monclient: init 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec1d76e0 con 0x7f22ec1430c0 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0abd80 con 0x7f22ec13f710 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec137c90 con 0x7f22ec0a98f0 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.141+0000 7f22f4486640 10 monclient(hunting): authenticate will time out at 5603.427716s 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.142+0000 7f22f2a23640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f22ec1430c0 0x7f22ec1e2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60824/0 (socket says 172.21.15.47:60824) 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.142+0000 7f22f2a23640 1 -- 172.21.15.47:0/1803170263 learned_addr learned my addr 172.21.15.47:0/1803170263 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.142+0000 7f22e2ffd640 10 client.?.objecter ms_handle_connect 0x7f22ec1430c0 2024-03-25T19:34:48.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.142+0000 7f22e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.145+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004487029 0 0) 0x7f22ec1d76e0 con 0x7f22ec1430c0 2024-03-25T19:34:48.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.145+0000 7f22e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.145+0000 7f22e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.145+0000 7f22e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.145+0000 7f22e2ffd640 10 monclient(hunting): my global_id is 4185 2024-03-25T19:34:48.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.146+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d0004eb0 con 0x7f22ec1430c0 2024-03-25T19:34:48.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.153+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542727145 0 0) 0x7f22d0004eb0 con 0x7f22ec1430c0 2024-03-25T19:34:48.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.154+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec1d76e0 con 0x7f22ec1430c0 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.154+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f22e4001ad0 con 0x7f22ec1430c0 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.154+0000 7f22e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.154+0000 7f22e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.154+0000 7f22e2ffd640 10 monclient(hunting): dump: 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.154+0000 7f22e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:48.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.156+0000 7f22e2ffd640 10 client.?.objecter ms_handle_connect 0x7f22ec0a98f0 2024-03-25T19:34:48.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.156+0000 7f22e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168142536 0 0) 0x7f22ec1d76e0 con 0x7f22ec1430c0 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 >> v1:172.21.15.47:6790/0 conn(0x7f22ec0a98f0 legacy=0x7f22ec1d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 >> v1:172.21.15.142:6789/0 conn(0x7f22ec13f710 legacy=0x7f22ec138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 1 monclient: found mon.a 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec1d41e0 con 0x7f22ec1430c0 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.159+0000 7f22e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 5 monclient: authenticate success, global_id 4185 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 10 monclient: _renew_subs 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec1d82d0 con 0x7f22ec1430c0 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 10 monclient: _renew_subs 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec138d80 con 0x7f22ec1430c0 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.160+0000 7f22f4486640 1 librados: init done 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.161+0000 7f22f4486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.161+0000 7f22f4486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.161+0000 7f22f4486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.161+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec0627b0 con 0x7f22ec1430c0 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e40020a0 con 0x7f22ec1430c0 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f22e4004d80 con 0x7f22ec1430c0 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22e2ffd640 10 monclient: dump: 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:48.631 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.631+0000 7f22f0a1f640 4 set_mon_vals no callback set 2024-03-25T19:34:48.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.633+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2119749153 0 0) 0x7f22e401ad60 con 0x7f22ec1430c0 2024-03-25T19:34:48.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.634+0000 7f22f2222640 10 monclient: get_auth_request con 0x7f22d006dab0 auth_method 0 2024-03-25T19:34:48.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.635+0000 7f22e2ffd640 10 client.4185.objecter ms_handle_connect 0x7f22d006dab0 2024-03-25T19:34:48.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f22e4054ab0 con 0x7f22ec1430c0 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 10 client.4185.objecter ms_dispatch 0x7f22ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 20 client.4185.objecter dump_active .. 0 homeless 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22e40503f0 con 0x7f22ec1430c0 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.644+0000 7f22e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:48.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:48.964+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f22e4050070 con 0x7f22ec1430c0 2024-03-25T19:34:49.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.008+0000 7f22f4486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:49.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.008+0000 7f22f4486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:49.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.008+0000 7f22f4486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:34:49.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.008+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22ec13b430 con 0x7f22ec1430c0 2024-03-25T19:34:49.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.108+0000 7f22e2ffd640 1 -- 172.21.15.47:0/1803170263 <== mon.0 v1:172.21.15.47:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f22e4004be0 con 0x7f22ec1430c0 2024-03-25T19:34:49.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.108+0000 7f22e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:49.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.108+0000 7f22e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:49.108 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:34:49.108 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:49.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.111+0000 7f22f4486640 10 librados: watch_flush enter 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.111+0000 7f22f4486640 10 librados: watch_flush exit 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.111+0000 7f22f4486640 20 client.4185.objecter shutdown clearing up homeless session... 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.111+0000 7f22f4486640 10 client.4185.objecter successfully canceled tick 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.112+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 >> v1:172.21.15.142:6800/1025302268 conn(0x7f22d006dab0 legacy=0x7f22d006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.112+0000 7f22f4486640 10 monclient: shutdown 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.112+0000 7f22f4486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.112+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 >> v1:172.21.15.47:6789/0 conn(0x7f22ec1430c0 legacy=0x7f22ec1e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.112+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 shutdown_connections 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.112+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 >> 172.21.15.47:0/1803170263 conn(0x7f22ec0b7ea0 msgr2=0x7f22ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:49.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.113+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 shutdown_connections 2024-03-25T19:34:49.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.113+0000 7f22f4486640 1 -- 172.21.15.47:0/1803170263 wait complete. 2024-03-25T19:34:49.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:49.113+0000 7f22f4486640 1 librados: shutdown 2024-03-25T19:34:49.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:53.254 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:06.747 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.316 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:06.809 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.327 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:06.815 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.388 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:06.876 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.397 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:06.891 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.458 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:06.952 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.481 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:06.975 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.545 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:07.039 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.815 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:07.292 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.822 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:07.299 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.851 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:07.342 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.864 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:07.336 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.867 INFO:tasks.ceph.osd.6.smithi142.stderr:2024-03-25T19:34:53.862+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:53.883 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:07.360 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.894 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:07.372 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.919 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:07.410 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.925 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:07.398 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:53.936 INFO:tasks.ceph.osd.7.smithi142.stderr:2024-03-25T19:34:53.930+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.005 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-03-25T19:34:54.000+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.113 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-03-25T19:34:54.107+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.322 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:07.815 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.388 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:07.876 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.454 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-25T19:34:54.437+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.458 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-25T19:34:54.454+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.476 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-25T19:34:54.470+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.478 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:07.972 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.493 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-25T19:34:54.486+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:34:54.584 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:08.079 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.689 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:08.182 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.756 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:08.244 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.862 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:08.356 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.916 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:08.393 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.949 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:08.443 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.953 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:08.430 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.961 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:08.452 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:54.968 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:08.441 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.008 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:08.501 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.034 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:08.512 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.101 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:08.589 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.214 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:08.708 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.265 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:08.760 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.283 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:08.776 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.307 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:08.798 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.338 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:08.815 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.342 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:08.814 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.357 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:08.834 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.367 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:08.855 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.379 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:08.870 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.509 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:09.003 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.535 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:09.029 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.550 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:09.043 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.621 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.099 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.635 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:09.123 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.652 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.143 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.663 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.135 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.702 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:09.179 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.799 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:09.293 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.803 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:09.297 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.821 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:09.314 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.887 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.364 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.899 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:09.387 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.916 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.407 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:55.931 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:55.937 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.410 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.006 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:09.483 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.021+0000 7f4111b97640 10 monclient: get_monmap_and_config 2024-03-25T19:34:56.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.021+0000 7f4111b97640 10 monclient: build_initial_monmap 2024-03-25T19:34:56.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.021+0000 7f4111b97640 10 monclient: monmap: 2024-03-25T19:34:56.020 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:34:56.022039+0000 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:34:56.022039+0000 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:34:56.021 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.021+0000 7f4111b97640 1 Processor -- start 2024-03-25T19:34:56.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 1 -- start start 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient: init 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient(hunting): picked mon.noname-c con 0x7f410c0a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient(hunting): picked mon.noname-b con 0x7f410c13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient(hunting): picked mon.noname-a con 0x7f410c1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c142520 con 0x7f410c1430c0 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0b28a0 con 0x7f410c0a98f0 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0b2a30 con 0x7f410c13f710 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.022+0000 7f4111b97640 10 monclient(hunting): authenticate will time out at 5611.308643s 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.023+0000 7f410affd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f410c13f710 0x7f410c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46804/0 (socket says 172.21.15.47:46804) 2024-03-25T19:34:56.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.023+0000 7f410affd640 1 -- 172.21.15.47:0/1402031552 learned_addr learned my addr 172.21.15.47:0/1402031552 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688798682 0 0) 0x7f410c0b2a30 con 0x7f410c13f710 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4003520 con 0x7f410c13f710 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665916695 0 0) 0x7f410c0b28a0 con 0x7f410c0a98f0 2024-03-25T19:34:56.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:56.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:56.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:56.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:34:56.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.025+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c0b2a30 con 0x7f410c0a98f0 2024-03-25T19:34:56.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.026+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171562506 0 0) 0x7f410c142520 con 0x7f410c1430c0 2024-03-25T19:34:56.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.026+0000 7f410a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:56.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.026+0000 7f410a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:56.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.026+0000 7f410a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:56.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.026+0000 7f410a7fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:34:56.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.026+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c0b28a0 con 0x7f410c1430c0 2024-03-25T19:34:56.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 314971369 0 0) 0x7f410c0b2a30 con 0x7f410c0a98f0 2024-03-25T19:34:56.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 >> v1:172.21.15.47:6790/0 conn(0x7f410c13f710 legacy=0x7f410c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 >> v1:172.21.15.47:6789/0 conn(0x7f410c1430c0 legacy=0x7f410c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:56.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0b44c0 con 0x7f410c0a98f0 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f40fc001670 con 0x7f410c0a98f0 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f410a7fc640 10 monclient: dump: 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.029 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f4111b97640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:34:56.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.029+0000 7f4111b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.030+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40fc0017d0 con 0x7f410c0a98f0 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f410a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f410a7fc640 1 -- 172.21.15.47:0/1402031552 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f40fc001be0 con 0x7f410c0a98f0 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 4 set_mon_vals no callback set 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f410a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f410a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f410a7fc640 10 monclient: dump: 2024-03-25T19:34:56.033 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 10 monclient: shutdown 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 1 -- 172.21.15.47:0/1402031552 >> v1:172.21.15.142:6789/0 conn(0x7f410c0a98f0 legacy=0x7f410c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 1 -- 172.21.15.47:0/1402031552 shutdown_connections 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 1 -- 172.21.15.47:0/1402031552 >> 172.21.15.47:0/1402031552 conn(0x7f410c0b7ea0 msgr2=0x7f410c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 1 -- 172.21.15.47:0/1402031552 shutdown_connections 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.031+0000 7f4111b97640 1 -- 172.21.15.47:0/1402031552 wait complete. 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.032+0000 7f4111b97640 10 monclient: build_initial_monmap 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.032+0000 7f4111b97640 10 monclient: monmap: 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:34:56.035 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.032+0000 7f4111b97640 1 librados: starting msgr at 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.032+0000 7f4111b97640 1 librados: starting objecter 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.032+0000 7f4111b97640 1 Processor -- start 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 1 -- start start 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 1 librados: setting wanted keys 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 1 librados: calling monclient init 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient: init 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient(hunting): picked mon.noname-a con 0x7f410c0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient(hunting): picked mon.noname-b con 0x7f410c13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient(hunting): picked mon.noname-c con 0x7f410c1430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c1390f0 con 0x7f410c0a98f0 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c138dd0 con 0x7f410c13f710 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c1346b0 con 0x7f410c1430c0 2024-03-25T19:34:56.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:56.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.033+0000 7f4111b97640 10 monclient(hunting): authenticate will time out at 5611.319568s 2024-03-25T19:34:56.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f410bfff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f410c1430c0 0x7f410c133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46810/0 (socket says 172.21.15.47:46810) 2024-03-25T19:34:56.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f410bfff640 1 -- 172.21.15.47:0/92192654 learned_addr learned my addr 172.21.15.47:0/92192654 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f40f3fff640 10 client.?.objecter ms_handle_connect 0x7f410c1430c0 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f40f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f40f3fff640 10 client.?.objecter ms_handle_connect 0x7f410c0a98f0 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f40f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f40f3fff640 10 client.?.objecter ms_handle_connect 0x7f410c13f710 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.034+0000 7f40f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:56.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195597736 0 0) 0x7f410c138dd0 con 0x7f410c13f710 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f8004eb0 con 0x7f410c13f710 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894338555 0 0) 0x7f410c1346b0 con 0x7f410c1430c0 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c138dd0 con 0x7f410c1430c0 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102671120 0 0) 0x7f410c1390f0 con 0x7f410c0a98f0 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:56.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:56.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:56.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:34:56.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.036+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c1346b0 con 0x7f410c0a98f0 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783761857 0 0) 0x7f410c138dd0 con 0x7f410c1430c0 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410c1390f0 con 0x7f410c1430c0 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4104001b90 con 0x7f410c1430c0 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): dump: 2024-03-25T19:34:56.042 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1736350875 0 0) 0x7f410c1346b0 con 0x7f410c0a98f0 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410c138dd0 con 0x7f410c0a98f0 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f40fc001ae0 con 0x7f410c0a98f0 2024-03-25T19:34:56.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): dump: 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:56.044 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3966059194 0 0) 0x7f40f8004eb0 con 0x7f410c13f710 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410c1346b0 con 0x7f410c13f710 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4100001950 con 0x7f410c13f710 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.042+0000 7f40f3fff640 10 monclient(hunting): dump: 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:56.045 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:56.046 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:56.046 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.045+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3765571233 0 0) 0x7f410c1346b0 con 0x7f410c13f710 2024-03-25T19:34:56.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.045+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 >> v1:172.21.15.47:6790/0 conn(0x7f410c1430c0 legacy=0x7f410c133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.045+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 >> v1:172.21.15.47:6789/0 conn(0x7f410c0a98f0 legacy=0x7f410c137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:56.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 1 monclient: found mon.b 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c134840 con 0x7f410c13f710 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f40f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 monclient: _renew_subs 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c134a90 con 0x7f410c13f710 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 monclient: _renew_subs 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:56.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c1356f0 con 0x7f410c13f710 2024-03-25T19:34:56.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 1 librados: init done 2024-03-25T19:34:56.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:56.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.046+0000 7f4111b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:56.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.047+0000 7f4111b97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:56.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.047+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c0627b0 con 0x7f410c13f710 2024-03-25T19:34:56.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4100001d60 con 0x7f410c13f710 2024-03-25T19:34:56.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f40f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:56.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4100004980 con 0x7f410c13f710 2024-03-25T19:34:56.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f40f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f40f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f40f3fff640 10 monclient: dump: 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:34:56.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.053+0000 7f4109ffb640 4 set_mon_vals no callback set 2024-03-25T19:34:56.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.061+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f410001a950 con 0x7f410c13f710 2024-03-25T19:34:56.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.062+0000 7f410b7fe640 10 monclient: get_auth_request con 0x7f40f806e130 auth_method 0 2024-03-25T19:34:56.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.062+0000 7f40f3fff640 10 client.4207.objecter ms_handle_connect 0x7f40f806e130 2024-03-25T19:34:56.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.063+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f4100054750 con 0x7f410c13f710 2024-03-25T19:34:56.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.063+0000 7f40f3fff640 10 client.4207.objecter ms_dispatch 0x7f410c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:56.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.063+0000 7f40f3fff640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:56.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.063+0000 7f40f3fff640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:56.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.063+0000 7f40f3fff640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:34:56.071 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:09.565 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.094 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:09.587 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.094 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:09.588 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.156 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.633 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.167 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:09.655 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.184 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.675 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.211 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.683 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.297 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:09.774 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.312+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4100050090 con 0x7f410c13f710 2024-03-25T19:34:56.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.312+0000 7f40f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:56.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.312+0000 7f40f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:56.335 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:09.829 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.369 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:09.863 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.370 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:09.863 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.420 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.898 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.431 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:09.919 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.452 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.943 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.476 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.949 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.493+0000 7f4111b97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.493+0000 7f4111b97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.493+0000 7f4111b97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:34:56.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.493+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f410c1352e0 con 0x7f410c13f710 2024-03-25T19:34:56.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.507+0000 7f40f3fff640 1 -- 172.21.15.47:0/92192654 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f410c1352e0 con 0x7f410c13f710 2024-03-25T19:34:56.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.507+0000 7f40f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.507+0000 7f40f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:56.507 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:34:56.507 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 10 librados: watch_flush enter 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 10 librados: watch_flush exit 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 >> v1:172.21.15.142:6800/1025302268 conn(0x7f40f806e130 legacy=0x7f40f80705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 10 monclient: shutdown 2024-03-25T19:34:56.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:56.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 >> v1:172.21.15.142:6789/0 conn(0x7f410c13f710 legacy=0x7f410c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 shutdown_connections 2024-03-25T19:34:56.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 >> 172.21.15.47:0/92192654 conn(0x7f410c0b7ea0 msgr2=0x7f410c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:56.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 shutdown_connections 2024-03-25T19:34:56.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 -- 172.21.15.47:0/92192654 wait complete. 2024-03-25T19:34:56.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:34:56.509+0000 7f4111b97640 1 librados: shutdown 2024-03-25T19:34:56.518 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:56.602 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.079 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.605 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:10.099 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.634 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:10.127 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.666 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:10.160 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.686 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.163 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.696 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:10.184 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.717 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.207 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.741 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.213 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.871 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:10.365 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.895 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.372 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.954 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.431 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.962 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:10.456 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:56.984 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.475 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.011 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.484 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.104 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:10.597 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.156 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:10.649 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.177 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:10.665 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.184 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.661 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.232 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:10.720 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.282 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.754 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.362 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:10.856 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.423 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:10.917 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.433 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.911 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.463 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.954 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.482 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.959 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.490 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.967 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.519 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:11.010 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.521 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:11.015 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.590 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:11.084 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.789 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:11.261 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:57.848 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:11.321 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:58.090 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:11.566 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:34:58.146 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:11.622 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:02.521 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:16.014 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:02.658 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:16.145 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:02.998 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:16.492 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:03.006 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:16.500 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:03.102 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:16.593 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:03.307 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:16.785 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:03.325 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.455+0000 7f1550e7b640 10 monclient: get_monmap_and_config 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.455+0000 7f1550e7b640 10 monclient: build_initial_monmap 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.455+0000 7f1550e7b640 10 monclient: monmap: 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:03.455798+0000 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:03.455798+0000 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.455+0000 7f1550e7b640 1 Processor -- start 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.455+0000 7f1550e7b640 1 -- start start 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.455+0000 7f1550e7b640 10 monclient: init 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f154c0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f154c0b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f154c14be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0af940 con 0x7f154c0b1730 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0b0a70 con 0x7f154c0b4d60 2024-03-25T19:35:03.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0b0d50 con 0x7f154c14be90 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.456+0000 7f1550e7b640 10 monclient(hunting): authenticate will time out at 5618.742458s 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.457+0000 7f154ad9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f154c14be90 0x7f154c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51614/0 (socket says 172.21.15.47:51614) 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.457+0000 7f154a59c640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f154c0b4d60 0x7f154c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:52420/0 (socket says 172.21.15.47:52420) 2024-03-25T19:35:03.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.457+0000 7f154ad9d640 1 -- 172.21.15.47:0/1722678358 learned_addr learned my addr 172.21.15.47:0/1722678358 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:03.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938033041 0 0) 0x7f154c0af940 con 0x7f154c0b1730 2024-03-25T19:35:03.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:35:03.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1534003520 con 0x7f154c0b1730 2024-03-25T19:35:03.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154936697 0 0) 0x7f154c0b0a70 con 0x7f154c0b4d60 2024-03-25T19:35:03.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.460+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f154c0af940 con 0x7f154c0b4d60 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.461+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526755285 0 0) 0x7f154c0b0d50 con 0x7f154c14be90 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.461+0000 7f154959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.461+0000 7f154959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.461+0000 7f154959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.461+0000 7f154959a640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.461+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f154c0b0a70 con 0x7f154c14be90 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3805511059 0 0) 0x7f1534003520 con 0x7f154c0b1730 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 >> v1:172.21.15.47:6790/0 conn(0x7f154c14be90 legacy=0x7f154c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 >> v1:172.21.15.142:6789/0 conn(0x7f154c0b4d60 legacy=0x7f154c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 10 monclient: _finish_hunting 0 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 1 monclient: found mon.noname-a 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.463+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0b1030 con 0x7f154c0b1730 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 10 monclient: _finish_auth 0 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 10 monclient: _check_auth_tickets 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1540001690 con 0x7f154c0b1730 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f154959a640 10 monclient: dump: 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f1550e7b640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:35:03.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.464+0000 7f1550e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:03.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15400018e0 con 0x7f154c0b1730 2024-03-25T19:35:03.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f154959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f154959a640 1 -- 172.21.15.47:0/1722678358 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1540001c70 con 0x7f154c0b1730 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f154959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f154959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f154959a640 10 monclient: dump: 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:03.470 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f1550e7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.465+0000 7f1550e7b640 4 set_mon_vals no callback set 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 10 monclient: shutdown 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 1 -- 172.21.15.47:0/1722678358 >> v1:172.21.15.47:6789/0 conn(0x7f154c0b1730 legacy=0x7f154c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 1 -- 172.21.15.47:0/1722678358 shutdown_connections 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 1 -- 172.21.15.47:0/1722678358 >> 172.21.15.47:0/1722678358 conn(0x7f154c0617a0 msgr2=0x7f154c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 1 -- 172.21.15.47:0/1722678358 shutdown_connections 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.466+0000 7f1550e7b640 1 -- 172.21.15.47:0/1722678358 wait complete. 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.467+0000 7f1550e7b640 10 monclient: build_initial_monmap 2024-03-25T19:35:03.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.467+0000 7f1550e7b640 10 monclient: monmap: 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 1 librados: starting msgr at 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 1 librados: starting objecter 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 1 Processor -- start 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 1 -- start start 2024-03-25T19:35:03.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 1 librados: setting wanted keys 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 1 librados: calling monclient init 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 10 monclient: init 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.468+0000 7f1550e7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f154c0b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f154c0b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f154c14be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c143d70 con 0x7f154c0b4d60 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c1451d0 con 0x7f154c14be90 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c1dce50 con 0x7f154c0b1730 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1550e7b640 10 monclient(hunting): authenticate will time out at 5618.755026s 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1549d9b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f154c0b4d60 0x7f154c1dc030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43760/0 (socket says 172.21.15.47:43760) 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.469+0000 7f1549d9b640 1 -- 172.21.15.47:0/2754471015 learned_addr learned my addr 172.21.15.47:0/2754471015 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:03.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.470+0000 7f153affd640 10 client.?.objecter ms_handle_connect 0x7f154c0b4d60 2024-03-25T19:35:03.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.470+0000 7f153affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.470+0000 7f153affd640 10 client.?.objecter ms_handle_connect 0x7f154c14be90 2024-03-25T19:35:03.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.470+0000 7f153affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.470+0000 7f153affd640 10 client.?.objecter ms_handle_connect 0x7f154c0b1730 2024-03-25T19:35:03.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.470+0000 7f153affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.472+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134594431 0 0) 0x7f154c143d70 con 0x7f154c0b4d60 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.472+0000 7f153affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.472+0000 7f153affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.472+0000 7f153affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.472+0000 7f153affd640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.472+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c004eb0 con 0x7f154c0b4d60 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020587492 0 0) 0x7f154c1451d0 con 0x7f154c14be90 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f154c143d70 con 0x7f154c14be90 2024-03-25T19:35:03.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978139887 0 0) 0x7f154c1dce50 con 0x7f154c0b1730 2024-03-25T19:35:03.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:35:03.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.473+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f154c1451d0 con 0x7f154c0b1730 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.479+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643575027 0 0) 0x7f154c143d70 con 0x7f154c14be90 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.479+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f154c1dce50 con 0x7f154c14be90 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1544001a50 con 0x7f154c14be90 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): dump: 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:03.479 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:03.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224236514 0 0) 0x7f154c1451d0 con 0x7f154c0b1730 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f154c143d70 con 0x7f154c0b1730 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f153c001930 con 0x7f154c0b1730 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 10 monclient(hunting): dump: 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:03.482 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.480+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781678694 0 0) 0x7f152c004eb0 con 0x7f154c0b4d60 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.481+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f154c1451d0 con 0x7f154c0b4d60 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.481+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1540001bf0 con 0x7f154c0b4d60 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.481+0000 7f153affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.481+0000 7f153affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.481+0000 7f153affd640 10 monclient(hunting): dump: 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.483 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:03.484 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.484+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2836330248 0 0) 0x7f154c1451d0 con 0x7f154c0b4d60 2024-03-25T19:35:03.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.484+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 >> v1:172.21.15.47:6790/0 conn(0x7f154c0b1730 legacy=0x7f154c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.484+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 >> v1:172.21.15.142:6789/0 conn(0x7f154c14be90 legacy=0x7f154c1dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 1 monclient: found mon.a 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c1454b0 con 0x7f154c0b4d60 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 10 monclient: _finish_auth 0 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f153affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 5 monclient: authenticate success, global_id 4221 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 10 monclient: _renew_subs 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c143900 con 0x7f154c0b4d60 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 10 monclient: _renew_subs 2024-03-25T19:35:03.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:35:03.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c13d9e0 con 0x7f154c0b4d60 2024-03-25T19:35:03.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.485+0000 7f1550e7b640 1 librados: init done 2024-03-25T19:35:03.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.487+0000 7f1550e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.487+0000 7f1550e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.487+0000 7f1550e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:35:03.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.487+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c1401f0 con 0x7f154c0b4d60 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1540001230 con 0x7f154c0b4d60 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1540003b00 con 0x7f154c0b4d60 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 10 monclient: dump: 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:03.511 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:03.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f1548d99640 4 set_mon_vals no callback set 2024-03-25T19:35:03.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.512+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f1540003d20 con 0x7f154c0b4d60 2024-03-25T19:35:03.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.513+0000 7f154a59c640 10 monclient: get_auth_request con 0x7f152c06e160 auth_method 0 2024-03-25T19:35:03.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.514+0000 7f153affd640 10 client.4221.objecter ms_handle_connect 0x7f152c06e160 2024-03-25T19:35:03.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.516+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f15400548c0 con 0x7f154c0b4d60 2024-03-25T19:35:03.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.516+0000 7f153affd640 10 client.4221.objecter ms_dispatch 0x7f154c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:03.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.516+0000 7f153affd640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:03.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.516+0000 7f153affd640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:03.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.517+0000 7f153affd640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T19:35:03.666 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:17.139 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:03.808 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:17.285 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.963+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15400503d0 con 0x7f154c0b4d60 2024-03-25T19:35:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.963+0000 7f153affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:03.963+0000 7f153affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:04.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.175+0000 7f1550e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.175+0000 7f1550e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.175+0000 7f1550e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:35:04.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.175+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f154c0627b0 con 0x7f154c0b4d60 2024-03-25T19:35:04.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.201+0000 7f153affd640 1 -- 172.21.15.47:0/2754471015 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f1540052300 con 0x7f154c0b4d60 2024-03-25T19:35:04.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.201+0000 7f153affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.201+0000 7f153affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:04.201 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:04.201 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:04.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.203+0000 7f1550e7b640 10 librados: watch_flush enter 2024-03-25T19:35:04.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.203+0000 7f1550e7b640 10 librados: watch_flush exit 2024-03-25T19:35:04.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.203+0000 7f1550e7b640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T19:35:04.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.203+0000 7f1550e7b640 10 client.4221.objecter successfully canceled tick 2024-03-25T19:35:04.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.204+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 >> v1:172.21.15.142:6800/1025302268 conn(0x7f152c06e160 legacy=0x7f152c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:04.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.204+0000 7f1550e7b640 10 monclient: shutdown 2024-03-25T19:35:04.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.204+0000 7f1550e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:04.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.204+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 >> v1:172.21.15.47:6789/0 conn(0x7f154c0b4d60 legacy=0x7f154c1dc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:04.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.205+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 shutdown_connections 2024-03-25T19:35:04.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.205+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 >> 172.21.15.47:0/2754471015 conn(0x7f154c0617a0 msgr2=0x7f154c0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:04.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.205+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 shutdown_connections 2024-03-25T19:35:04.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.205+0000 7f1550e7b640 1 -- 172.21.15.47:0/2754471015 wait complete. 2024-03-25T19:35:04.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:04.205+0000 7f1550e7b640 1 librados: shutdown 2024-03-25T19:35:04.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:08.145 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:21.633 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.149 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:21.637 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.229 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:21.722 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.234 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:21.727 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.869 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:22.363 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.875 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:22.369 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.880 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:22.374 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:08.883 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:22.378 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.019 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:22.497 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.023 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:22.501 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.304 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:22.795 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.308 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:22.799 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.576 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:23.048 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.581 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:23.054 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.863 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:23.340 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:09.869 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:23.346 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:11.023 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.157+0000 7f48b1e0f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.157+0000 7f48b1e0f640 10 monclient: build_initial_monmap 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.157+0000 7f48b1e0f640 10 monclient: monmap: 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:11.159397+0000 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:11.159397+0000 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:11.158 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.158+0000 7f48b1e0f640 1 Processor -- start 2024-03-25T19:35:11.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.158+0000 7f48b1e0f640 1 -- start start 2024-03-25T19:35:11.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.158+0000 7f48b1e0f640 10 monclient: init 2024-03-25T19:35:11.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:11.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:11.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f48ac0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f48ac13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f48ac1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac142520 con 0x7f48ac13f710 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.159+0000 7f48b1e0f640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac0b28a0 con 0x7f48ac1430c0 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.160+0000 7f48b1e0f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac0b2a30 con 0x7f48ac0a98f0 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.160+0000 7f48b1e0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.160+0000 7f48b1e0f640 10 monclient(hunting): authenticate will time out at 5626.447014s 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.160+0000 7f48abfff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f48ac1430c0 0x7f48ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:39964/0 (socket says 172.21.15.47:39964) 2024-03-25T19:35:11.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.160+0000 7f48abfff640 1 -- 172.21.15.47:0/1193742934 learned_addr learned my addr 172.21.15.47:0/1193742934 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:11.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.163+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880118907 0 0) 0x7f48ac0b28a0 con 0x7f48ac1430c0 2024-03-25T19:35:11.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.163+0000 7f48aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.163+0000 7f48aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.163+0000 7f48aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.163+0000 7f48aa7fc640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:35:11.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.163+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4894003520 con 0x7f48ac1430c0 2024-03-25T19:35:11.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541648986 0 0) 0x7f48ac0b2a30 con 0x7f48ac0a98f0 2024-03-25T19:35:11.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac0b28a0 con 0x7f48ac0a98f0 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070358348 0 0) 0x7f48ac142520 con 0x7f48ac13f710 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:35:11.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.164+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac0b2a30 con 0x7f48ac13f710 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207073131 0 0) 0x7f4894003520 con 0x7f48ac1430c0 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 >> v1:172.21.15.47:6790/0 conn(0x7f48ac0a98f0 legacy=0x7f48ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 >> v1:172.21.15.47:6789/0 conn(0x7f48ac13f710 legacy=0x7f48ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 1 monclient: found mon.noname-c 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0b44c0 con 0x7f48ac1430c0 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:11.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f48a4001a10 con 0x7f48ac1430c0 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48aa7fc640 10 monclient: dump: 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48b1e0f640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:35:11.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.168+0000 7f48b1e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48a4001b70 con 0x7f48ac1430c0 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48aa7fc640 1 -- 172.21.15.47:0/1193742934 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f48a4001f80 con 0x7f48ac1430c0 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48aa7fc640 10 monclient: dump: 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:11.173 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48b1e0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.171+0000 7f48b1e0f640 4 set_mon_vals no callback set 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 10 monclient: shutdown 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 1 -- 172.21.15.47:0/1193742934 >> v1:172.21.15.142:6789/0 conn(0x7f48ac1430c0 legacy=0x7f48ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 1 -- 172.21.15.47:0/1193742934 shutdown_connections 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 1 -- 172.21.15.47:0/1193742934 >> 172.21.15.47:0/1193742934 conn(0x7f48ac0b7ea0 msgr2=0x7f48ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 1 -- 172.21.15.47:0/1193742934 shutdown_connections 2024-03-25T19:35:11.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.172+0000 7f48b1e0f640 1 -- 172.21.15.47:0/1193742934 wait complete. 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.173+0000 7f48b1e0f640 10 monclient: build_initial_monmap 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.173+0000 7f48b1e0f640 10 monclient: monmap: 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:11.177 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 1 librados: starting msgr at 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 1 librados: starting objecter 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 1 Processor -- start 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 1 -- start start 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 1 librados: setting wanted keys 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 1 librados: calling monclient init 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.174+0000 7f48b1e0f640 10 monclient: init 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.175+0000 7f48b1e0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.175+0000 7f48b1e0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.175+0000 7f48b1e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f48ac0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.175+0000 7f48b1e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f48ac13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48b1e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f48ac1430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48b1e0f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac135230 con 0x7f48ac0a98f0 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48b1e0f640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac135da0 con 0x7f48ac13f710 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48b1e0f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac135f30 con 0x7f48ac1430c0 2024-03-25T19:35:11.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48b1e0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48b1e0f640 10 monclient(hunting): authenticate will time out at 5626.463179s 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48aaffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f48ac13f710 0x7f48ac1dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:39970/0 (socket says 172.21.15.47:39970) 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f48aaffd640 1 -- 172.21.15.47:0/2091385002 learned_addr learned my addr 172.21.15.47:0/2091385002 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f4893fff640 10 client.?.objecter ms_handle_connect 0x7f48ac13f710 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.176+0000 7f4893fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.177+0000 7f4893fff640 10 client.?.objecter ms_handle_connect 0x7f48ac0a98f0 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.177+0000 7f4893fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.177+0000 7f4893fff640 10 client.?.objecter ms_handle_connect 0x7f48ac1430c0 2024-03-25T19:35:11.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.177+0000 7f4893fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575322184 0 0) 0x7f48ac135f30 con 0x7f48ac1430c0 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4898004eb0 con 0x7f48ac1430c0 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627421094 0 0) 0x7f48ac135da0 con 0x7f48ac13f710 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:35:11.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.179+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac135f30 con 0x7f48ac13f710 2024-03-25T19:35:11.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.180+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286760798 0 0) 0x7f48ac135230 con 0x7f48ac0a98f0 2024-03-25T19:35:11.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.180+0000 7f4893fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.180+0000 7f4893fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.180+0000 7f4893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.180+0000 7f4893fff640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:35:11.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.180+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac135da0 con 0x7f48ac0a98f0 2024-03-25T19:35:11.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.186+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663810565 0 0) 0x7f48ac135f30 con 0x7f48ac13f710 2024-03-25T19:35:11.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.186+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ac135230 con 0x7f48ac13f710 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126967854 0 0) 0x7f48ac135da0 con 0x7f48ac0a98f0 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ac135f30 con 0x7f48ac0a98f0 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2609521944 0 0) 0x7f4898004eb0 con 0x7f48ac1430c0 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ac135da0 con 0x7f48ac1430c0 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f48a0001a30 con 0x7f48ac13f710 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): dump: 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.188 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f48a4001e80 con 0x7f48ac1430c0 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.187+0000 7f4893fff640 10 monclient(hunting): dump: 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:11.189 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:11.190 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:11.190 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2950510414 0 0) 0x7f48ac135230 con 0x7f48ac13f710 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 >> v1:172.21.15.47:6790/0 conn(0x7f48ac1430c0 legacy=0x7f48ac1db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 >> v1:172.21.15.47:6789/0 conn(0x7f48ac0a98f0 legacy=0x7f48ac1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 1 monclient: found mon.b 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.191+0000 7f4893fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac1360c0 con 0x7f48ac13f710 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f4893fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f4893fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f4893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:11.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 10 monclient: _renew_subs 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac1d5000 con 0x7f48ac13f710 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 10 monclient: _renew_subs 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac1d41e0 con 0x7f48ac13f710 2024-03-25T19:35:11.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.192+0000 7f48b1e0f640 1 librados: init done 2024-03-25T19:35:11.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.193+0000 7f48b1e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.193+0000 7f48b1e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.193+0000 7f48b1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:11.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.193+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac1d4530 con 0x7f48ac13f710 2024-03-25T19:35:11.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.212+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48a0001b90 con 0x7f48ac13f710 2024-03-25T19:35:11.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.212+0000 7f4893fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.212+0000 7f48a9ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.213+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f48a0003a60 con 0x7f48ac13f710 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.213+0000 7f4893fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.213+0000 7f4893fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.213+0000 7f4893fff640 10 monclient: dump: 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:11.214 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:11.215 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:11.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.213+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f48a0003c80 con 0x7f48ac13f710 2024-03-25T19:35:11.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.214+0000 7f48ab7fe640 10 monclient: get_auth_request con 0x7f489806e010 auth_method 0 2024-03-25T19:35:11.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.214+0000 7f4893fff640 10 client.4219.objecter ms_handle_connect 0x7f489806e010 2024-03-25T19:35:11.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.214+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f48a0054920 con 0x7f48ac13f710 2024-03-25T19:35:11.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.215+0000 7f4893fff640 10 client.4219.objecter ms_dispatch 0x7f48ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:11.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.215+0000 7f4893fff640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:11.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.215+0000 7f4893fff640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:11.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.215+0000 7f4893fff640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T19:35:11.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.511+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48a0050260 con 0x7f48ac13f710 2024-03-25T19:35:11.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.511+0000 7f4893fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.511+0000 7f4893fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:11.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.836+0000 7f48b1e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.836+0000 7f48b1e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.836+0000 7f48b1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:11.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.836+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48ac135090 con 0x7f48ac13f710 2024-03-25T19:35:11.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.860+0000 7f4893fff640 1 -- 172.21.15.47:0/2091385002 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f48ac135090 con 0x7f48ac13f710 2024-03-25T19:35:11.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.861+0000 7f4893fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.861+0000 7f4893fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:11.861 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:11.861 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 10 librados: watch_flush enter 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 10 librados: watch_flush exit 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 10 client.4219.objecter successfully canceled tick 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 >> v1:172.21.15.142:6800/1025302268 conn(0x7f489806e010 legacy=0x7f48980704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 10 monclient: shutdown 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 >> v1:172.21.15.142:6789/0 conn(0x7f48ac13f710 legacy=0x7f48ac1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 shutdown_connections 2024-03-25T19:35:11.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 >> 172.21.15.47:0/2091385002 conn(0x7f48ac0b7ea0 msgr2=0x7f48ac14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.863+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 shutdown_connections 2024-03-25T19:35:11.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.864+0000 7f48b1e0f640 1 -- 172.21.15.47:0/2091385002 wait complete. 2024-03-25T19:35:11.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:11.864+0000 7f48b1e0f640 1 librados: shutdown 2024-03-25T19:35:11.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:11.979 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:00:25.467 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:12.095 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:00:25.588 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:12.886 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:00:26.380 35553== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:12.987 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:00:26.481 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:13.255 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:26.732 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:13.739 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:27.229 35558== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:13.920 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:27.393 35561== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:14.435 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:27.912 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:18.683 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:18.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.793+0000 7fd98b7f6640 10 monclient: get_monmap_and_config 2024-03-25T19:35:18.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.793+0000 7fd98b7f6640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.793+0000 7fd98b7f6640 10 monclient: monmap: 2024-03-25T19:35:18.793 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:18.794 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:18.794 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:18.794645+0000 2024-03-25T19:35:18.794 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:18.794645+0000 2024-03-25T19:35:18.794 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.794 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:18.794 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:18.795 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:18.795 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:18.795 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.794+0000 7fd98b7f6640 1 Processor -- start 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.795+0000 7fd98b7f6640 1 -- start start 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.795+0000 7fd98b7f6640 10 monclient: init 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.795+0000 7fd98b7f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.795+0000 7fd98b7f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.795+0000 7fd98b7f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd9840a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd98413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:18.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd9841430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd984142520 con 0x7fd9840a98f0 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9840b28a0 con 0x7fd98413f710 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9840b2a30 con 0x7fd9841430c0 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd98b7f6640 10 monclient(hunting): authenticate will time out at 5634.082276s 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd989d93640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd9841430c0 0x7fd98414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44080/0 (socket says 172.21.15.47:44080) 2024-03-25T19:35:18.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.796+0000 7fd989d93640 1 -- 172.21.15.47:0/713364591 learned_addr learned my addr 172.21.15.47:0/713364591 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:18.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472737210 0 0) 0x7fd9840b2a30 con 0x7fd9841430c0 2024-03-25T19:35:18.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd970003520 con 0x7fd9841430c0 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421228772 0 0) 0x7fd984142520 con 0x7fd9840a98f0 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 10 monclient(hunting): my global_id is 4230 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.799+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9840b2a30 con 0x7fd9840a98f0 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.800+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878543224 0 0) 0x7fd9840b28a0 con 0x7fd98413f710 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.800+0000 7fd97bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.800+0000 7fd97bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.800+0000 7fd97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.800+0000 7fd97bfff640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:35:18.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.800+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984142520 con 0x7fd98413f710 2024-03-25T19:35:18.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.802+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600785577 0 0) 0x7fd970003520 con 0x7fd9841430c0 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.802+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 >> v1:172.21.15.142:6789/0 conn(0x7fd98413f710 legacy=0x7fd9841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.802+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 >> v1:172.21.15.47:6789/0 conn(0x7fd9840a98f0 legacy=0x7fd9840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.802+0000 7fd97bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.802+0000 7fd97bfff640 1 monclient: found mon.noname-b 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.802+0000 7fd97bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9840b44c0 con 0x7fd9841430c0 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd97c001a10 con 0x7fd9841430c0 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd97bfff640 10 monclient: dump: 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:18.804 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:18.805 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd98b7f6640 5 monclient: authenticate success, global_id 4217 2024-03-25T19:35:18.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.803+0000 7fd98b7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd97c001b70 con 0x7fd9841430c0 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd97bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd97bfff640 1 -- 172.21.15.47:0/713364591 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd97c001f80 con 0x7fd9841430c0 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd98b7f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd98b7f6640 4 set_mon_vals no callback set 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd97bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd97bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.804+0000 7fd97bfff640 10 monclient: dump: 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.805+0000 7fd98b7f6640 10 monclient: shutdown 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.805+0000 7fd98b7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.805+0000 7fd98b7f6640 1 -- 172.21.15.47:0/713364591 >> v1:172.21.15.47:6790/0 conn(0x7fd9841430c0 legacy=0x7fd98414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.806+0000 7fd98b7f6640 1 -- 172.21.15.47:0/713364591 shutdown_connections 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.806+0000 7fd98b7f6640 1 -- 172.21.15.47:0/713364591 >> 172.21.15.47:0/713364591 conn(0x7fd9840b7ea0 msgr2=0x7fd9840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.807+0000 7fd98b7f6640 1 -- 172.21.15.47:0/713364591 shutdown_connections 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.807+0000 7fd98b7f6640 1 -- 172.21.15.47:0/713364591 wait complete. 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 10 monclient: monmap: 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:18.809 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:18.810 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.810 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:18.810 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:18.810 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:18.810 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:18.810 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 1 librados: starting msgr at 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 1 librados: starting objecter 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 1 Processor -- start 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 1 -- start start 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 1 librados: setting wanted keys 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 1 librados: calling monclient init 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.808+0000 7fd98b7f6640 10 monclient: init 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd9840a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd98413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd9841430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9841396e0 con 0x7fd9840a98f0 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.809+0000 7fd98b7f6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd984138530 con 0x7fd98413f710 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd989d93640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd9841430c0 0x7fd9841e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44092/0 (socket says 172.21.15.47:44092) 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd989d93640 1 -- 172.21.15.47:0/2392022894 learned_addr learned my addr 172.21.15.47:0/2392022894 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9841399c0 con 0x7fd9841430c0 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd98b7f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd98b7f6640 10 monclient(hunting): authenticate will time out at 5634.096079s 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd979ffb640 10 client.?.objecter ms_handle_connect 0x7fd9841430c0 2024-03-25T19:35:18.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd979ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd979ffb640 10 client.?.objecter ms_handle_connect 0x7fd9840a98f0 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd979ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd979ffb640 10 client.?.objecter ms_handle_connect 0x7fd98413f710 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.810+0000 7fd979ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.812+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228624351 0 0) 0x7fd9841399c0 con 0x7fd9841430c0 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.812+0000 7fd979ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.812+0000 7fd979ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.812+0000 7fd979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.812+0000 7fd979ffb640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.812+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968004eb0 con 0x7fd9841430c0 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.813+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11902001 0 0) 0x7fd984138530 con 0x7fd98413f710 2024-03-25T19:35:18.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.813+0000 7fd979ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.813+0000 7fd979ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.813+0000 7fd979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.813+0000 7fd979ffb640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.813+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9841399c0 con 0x7fd98413f710 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.814+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023679442 0 0) 0x7fd9841396e0 con 0x7fd9840a98f0 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.814+0000 7fd979ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.814+0000 7fd979ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.814+0000 7fd979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.814+0000 7fd979ffb640 10 monclient(hunting): my global_id is 4233 2024-03-25T19:35:18.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.814+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984138530 con 0x7fd9840a98f0 2024-03-25T19:35:18.826 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33037953 0 0) 0x7fd968004eb0 con 0x7fd9841430c0 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9841396e0 con 0x7fd9841430c0 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd97c001ec0 con 0x7fd9841430c0 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:18.827 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.817+0000 7fd979ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.820+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278983638 0 0) 0x7fd9841399c0 con 0x7fd98413f710 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.820+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd968004eb0 con 0x7fd98413f710 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.820+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd974001930 con 0x7fd98413f710 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.820+0000 7fd979ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.820+0000 7fd979ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.820+0000 7fd979ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:18.828 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3557358245 0 0) 0x7fd9841396e0 con 0x7fd9841430c0 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 >> v1:172.21.15.142:6789/0 conn(0x7fd98413f710 legacy=0x7fd9841da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 >> v1:172.21.15.47:6789/0 conn(0x7fd9840a98f0 legacy=0x7fd984137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 1 monclient: found mon.c 2024-03-25T19:35:18.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984138a50 con 0x7fd9841430c0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.821+0000 7fd979ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 5 monclient: authenticate success, global_id 4220 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 10 monclient: _renew_subs 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9841380a0 con 0x7fd9841430c0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 10 monclient: _renew_subs 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9841db140 con 0x7fd9841430c0 2024-03-25T19:35:18.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.822+0000 7fd98b7f6640 1 librados: init done 2024-03-25T19:35:18.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.823+0000 7fd98b7f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.823+0000 7fd98b7f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.823+0000 7fd98b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:18.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.823+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd984138ca0 con 0x7fd9841430c0 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd97c004e10 con 0x7fd9841430c0 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd979ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd97c0015d0 con 0x7fd9841430c0 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd979ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd979ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd979ffb640 10 monclient: dump: 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.840 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.840+0000 7fd97b7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.841+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fd97c01adf0 con 0x7fd9841430c0 2024-03-25T19:35:18.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.842+0000 7fd989592640 10 monclient: get_auth_request con 0x7fd96806dea0 auth_method 0 2024-03-25T19:35:18.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.842+0000 7fd979ffb640 10 client.4220.objecter ms_handle_connect 0x7fd96806dea0 2024-03-25T19:35:18.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.844+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fd97c054a40 con 0x7fd9841430c0 2024-03-25T19:35:18.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.844+0000 7fd979ffb640 10 client.4220.objecter ms_dispatch 0x7fd984000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:18.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.844+0000 7fd979ffb640 3 client.4220.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:18.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.844+0000 7fd979ffb640 3 client.4220.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:18.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:18.844+0000 7fd979ffb640 20 client.4220.objecter dump_active .. 0 homeless 2024-03-25T19:35:19.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.201+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd97c050550 con 0x7fd9841430c0 2024-03-25T19:35:19.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.201+0000 7fd979ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:19.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.201+0000 7fd979ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:19.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.445+0000 7fd98b7f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.445+0000 7fd98b7f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.445+0000 7fd98b7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:19.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.445+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9840627b0 con 0x7fd9841430c0 2024-03-25T19:35:19.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.533+0000 7fd979ffb640 1 -- 172.21.15.47:0/2392022894 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fd97c052480 con 0x7fd9841430c0 2024-03-25T19:35:19.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.533+0000 7fd979ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.533+0000 7fd979ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:19.533 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:19.533 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 10 librados: watch_flush enter 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 10 librados: watch_flush exit 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 20 client.4220.objecter shutdown clearing up homeless session... 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 10 client.4220.objecter successfully canceled tick 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 >> v1:172.21.15.142:6800/1025302268 conn(0x7fd96806dea0 legacy=0x7fd968070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 10 monclient: shutdown 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 >> v1:172.21.15.47:6790/0 conn(0x7fd9841430c0 legacy=0x7fd9841e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.536+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 shutdown_connections 2024-03-25T19:35:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.537+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 >> 172.21.15.47:0/2392022894 conn(0x7fd9840b7ea0 msgr2=0x7fd984074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.537+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 shutdown_connections 2024-03-25T19:35:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.537+0000 7fd98b7f6640 1 -- 172.21.15.47:0/2392022894 wait complete. 2024-03-25T19:35:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:19.537+0000 7fd98b7f6640 1 librados: shutdown 2024-03-25T19:35:19.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:20.253 INFO:tasks.ceph.osd.6.smithi142.stderr:2024-03-25T19:35:20.249+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:35:20.283 INFO:tasks.ceph.osd.7.smithi142.stderr:2024-03-25T19:35:20.281+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:35:20.814 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-03-25T19:35:20.802+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:35:21.168 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-03-25T19:35:21.165+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:35:22.364 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-25T19:35:22.361+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:35:22.897 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-25T19:35:22.894+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:35:22.961 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-25T19:35:22.958+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:35:23.620 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-25T19:35:23.617+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:35:26.353 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.463+0000 7f1c5000f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.463+0000 7f1c5000f640 10 monclient: build_initial_monmap 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.463+0000 7f1c5000f640 10 monclient: monmap: 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:26.464399+0000 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:26.464399+0000 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:26.463 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:26.464 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:26.464 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.464+0000 7f1c5000f640 1 Processor -- start 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.464+0000 7f1c5000f640 1 -- start start 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.464+0000 7f1c5000f640 10 monclient: init 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c480a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c481430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c48142520 con 0x7f1c481430c0 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480b28a0 con 0x7f1c4813f710 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480b2a30 con 0x7f1c480a98f0 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:26.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.465+0000 7f1c5000f640 10 monclient(hunting): authenticate will time out at 5641.751220s 2024-03-25T19:35:26.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.478+0000 7f1c4e5ac640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1c481430c0 0x7f1c4814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55314/0 (socket says 172.21.15.47:55314) 2024-03-25T19:35:26.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.478+0000 7f1c4e5ac640 1 -- 172.21.15.47:0/2311190908 learned_addr learned my addr 172.21.15.47:0/2311190908 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:26.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.498+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088960183 0 0) 0x7f1c480b2a30 con 0x7f1c480a98f0 2024-03-25T19:35:26.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.498+0000 7f1c4cda9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.498+0000 7f1c4cda9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.498+0000 7f1c4cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.498+0000 7f1c4cda9640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:35:26.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.498+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c34003520 con 0x7f1c480a98f0 2024-03-25T19:35:26.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.502+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190046979 0 0) 0x7f1c480b28a0 con 0x7f1c4813f710 2024-03-25T19:35:26.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.502+0000 7f1c4cda9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.502+0000 7f1c4cda9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.502+0000 7f1c4cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.502+0000 7f1c4cda9640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.502+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c480b2a30 con 0x7f1c4813f710 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.503+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22240199 0 0) 0x7f1c48142520 con 0x7f1c481430c0 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.503+0000 7f1c4cda9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.503+0000 7f1c4cda9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.503+0000 7f1c4cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.503+0000 7f1c4cda9640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:35:26.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.503+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c480b28a0 con 0x7f1c481430c0 2024-03-25T19:35:26.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 345443258 0 0) 0x7f1c480b2a30 con 0x7f1c4813f710 2024-03-25T19:35:26.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 >> v1:172.21.15.47:6790/0 conn(0x7f1c480a98f0 legacy=0x7f1c480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 >> v1:172.21.15.47:6789/0 conn(0x7f1c481430c0 legacy=0x7f1c4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 1 monclient: found mon.noname-c 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480b44c0 con 0x7f1c4813f710 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: _finish_auth 0 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1c38001670 con 0x7f1c4813f710 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c4cda9640 10 monclient: dump: 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.513 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c5000f640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:35:26.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.513+0000 7f1c5000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c380017d0 con 0x7f1c4813f710 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c4cda9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c4cda9640 1 -- 172.21.15.47:0/2311190908 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1c38001be0 con 0x7f1c4813f710 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c4cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c4cda9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c4cda9640 10 monclient: dump: 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.519 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c5000f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:26.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.519+0000 7f1c5000f640 4 set_mon_vals no callback set 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.520+0000 7f1c5000f640 10 monclient: shutdown 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.520+0000 7f1c5000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.520+0000 7f1c5000f640 1 -- 172.21.15.47:0/2311190908 >> v1:172.21.15.142:6789/0 conn(0x7f1c4813f710 legacy=0x7f1c481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.520+0000 7f1c5000f640 1 -- 172.21.15.47:0/2311190908 shutdown_connections 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.520+0000 7f1c5000f640 1 -- 172.21.15.47:0/2311190908 >> 172.21.15.47:0/2311190908 conn(0x7f1c480b7ea0 msgr2=0x7f1c480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.521+0000 7f1c5000f640 1 -- 172.21.15.47:0/2311190908 shutdown_connections 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.521+0000 7f1c5000f640 1 -- 172.21.15.47:0/2311190908 wait complete. 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.521+0000 7f1c5000f640 10 monclient: build_initial_monmap 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.521+0000 7f1c5000f640 10 monclient: monmap: 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:26.522 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 1 librados: starting msgr at 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 1 librados: starting objecter 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 1 Processor -- start 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 1 -- start start 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 1 librados: setting wanted keys 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 1 librados: calling monclient init 2024-03-25T19:35:26.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.522+0000 7f1c5000f640 10 monclient: init 2024-03-25T19:35:26.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:26.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:26.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c480a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:26.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:26.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c481430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:26.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480abde0 con 0x7f1c481430c0 2024-03-25T19:35:26.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480ac0c0 con 0x7f1c4813f710 2024-03-25T19:35:26.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c48139250 con 0x7f1c480a98f0 2024-03-25T19:35:26.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:26.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.523+0000 7f1c5000f640 10 monclient(hunting): authenticate will time out at 5641.809313s 2024-03-25T19:35:26.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.530+0000 7f1c4e5ac640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1c481430c0 0x7f1c481e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55348/0 (socket says 172.21.15.47:55348) 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.530+0000 7f1c4e5ac640 1 -- 172.21.15.47:0/4274951741 learned_addr learned my addr 172.21.15.47:0/4274951741 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.531+0000 7f1c3e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c481430c0 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.531+0000 7f1c3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.537+0000 7f1c3e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c4813f710 2024-03-25T19:35:26.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.537+0000 7f1c3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.543+0000 7f1c3e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c480a98f0 2024-03-25T19:35:26.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.543+0000 7f1c3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.547+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196072416 0 0) 0x7f1c480abde0 con 0x7f1c481430c0 2024-03-25T19:35:26.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.547+0000 7f1c3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.547+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.547+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.547+0000 7f1c3e7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:35:26.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.547+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c004eb0 con 0x7f1c481430c0 2024-03-25T19:35:26.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.552+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340416850 0 0) 0x7f1c48139250 con 0x7f1c480a98f0 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.552+0000 7f1c3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.552+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.552+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.552+0000 7f1c3e7fc640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.552+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c480abde0 con 0x7f1c480a98f0 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.553+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222934384 0 0) 0x7f1c480ac0c0 con 0x7f1c4813f710 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.553+0000 7f1c3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.553+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.553+0000 7f1c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.553+0000 7f1c3e7fc640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:35:26.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.553+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c48139250 con 0x7f1c4813f710 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.563+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613117421 0 0) 0x7f1c2c004eb0 con 0x7f1c481430c0 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.563+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c480ac0c0 con 0x7f1c481430c0 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.563+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1c40001bd0 con 0x7f1c481430c0 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.563+0000 7f1c3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.564+0000 7f1c3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.564+0000 7f1c3e7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.563 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.564+0000 7f1c3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.567+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3628533942 0 0) 0x7f1c480abde0 con 0x7f1c480a98f0 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.567+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c2c004eb0 con 0x7f1c480a98f0 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.567+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1c44001950 con 0x7f1c480a98f0 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.567+0000 7f1c3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.567+0000 7f1c3e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.567+0000 7f1c3e7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:26.567 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:26.568 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.572+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 750376287 0 0) 0x7f1c48139250 con 0x7f1c4813f710 2024-03-25T19:35:26.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.572+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c480abde0 con 0x7f1c4813f710 2024-03-25T19:35:26.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.572+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1c38001ae0 con 0x7f1c4813f710 2024-03-25T19:35:26.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.572+0000 7f1c3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.572+0000 7f1c3e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.572+0000 7f1c3e7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:26.572 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2708556830 0 0) 0x7f1c480abde0 con 0x7f1c4813f710 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 >> v1:172.21.15.47:6790/0 conn(0x7f1c480a98f0 legacy=0x7f1c48138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 >> v1:172.21.15.47:6789/0 conn(0x7f1c481430c0 legacy=0x7f1c481e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 1 monclient: found mon.b 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480acbf0 con 0x7f1c4813f710 2024-03-25T19:35:26.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 10 monclient: _renew_subs 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c48137de0 con 0x7f1c4813f710 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 10 monclient: _renew_subs 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c481368c0 con 0x7f1c4813f710 2024-03-25T19:35:26.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.579+0000 7f1c5000f640 1 librados: init done 2024-03-25T19:35:26.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.580+0000 7f1c5000f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.580+0000 7f1c5000f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.580+0000 7f1c5000f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:26.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.580+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48136bc0 con 0x7f1c4813f710 2024-03-25T19:35:26.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.854+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c38001210 con 0x7f1c4813f710 2024-03-25T19:35:26.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.854+0000 7f1c3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.854+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1c380038a0 con 0x7f1c4813f710 2024-03-25T19:35:26.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.854+0000 7f1c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.855+0000 7f1c3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.855+0000 7f1c3e7fc640 10 monclient: dump: 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:26.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.854+0000 7f1c3ffff640 4 set_mon_vals no callback set 2024-03-25T19:35:26.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.855+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f1c38019870 con 0x7f1c4813f710 2024-03-25T19:35:26.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.856+0000 7f1c4ddab640 10 monclient: get_auth_request con 0x7f1c2c06e110 auth_method 0 2024-03-25T19:35:26.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.856+0000 7f1c3e7fc640 10 client.4255.objecter ms_handle_connect 0x7f1c2c06e110 2024-03-25T19:35:26.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.860+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f1c380534e0 con 0x7f1c4813f710 2024-03-25T19:35:26.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.860+0000 7f1c3e7fc640 10 client.4255.objecter ms_dispatch 0x7f1c48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:26.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.860+0000 7f1c3e7fc640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:26.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.860+0000 7f1c3e7fc640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:26.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.860+0000 7f1c3e7fc640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T19:35:26.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.861+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c3804ee20 con 0x7f1c4813f710 2024-03-25T19:35:26.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.861+0000 7f1c3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:26.861+0000 7f1c3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.117+0000 7f1c5000f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.117+0000 7f1c5000f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.117+0000 7f1c5000f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.117+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c4814f2d0 con 0x7f1c4813f710 2024-03-25T19:35:27.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.153+0000 7f1c3e7fc640 1 -- 172.21.15.47:0/4274951741 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f1c38050f20 con 0x7f1c4813f710 2024-03-25T19:35:27.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.153+0000 7f1c3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:27.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.153+0000 7f1c3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:27.153 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:27.153 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:27.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 10 librados: watch_flush enter 2024-03-25T19:35:27.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 10 librados: watch_flush exit 2024-03-25T19:35:27.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T19:35:27.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 10 client.4255.objecter successfully canceled tick 2024-03-25T19:35:27.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 >> v1:172.21.15.142:6800/1025302268 conn(0x7f1c2c06e110 legacy=0x7f1c2c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 10 monclient: shutdown 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 >> v1:172.21.15.142:6789/0 conn(0x7f1c4813f710 legacy=0x7f1c480ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 shutdown_connections 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.156+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 >> 172.21.15.47:0/4274951741 conn(0x7f1c480b7ea0 msgr2=0x7f1c48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.157+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 shutdown_connections 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.157+0000 7f1c5000f640 1 -- 172.21.15.47:0/4274951741 wait complete. 2024-03-25T19:35:27.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:27.157+0000 7f1c5000f640 1 librados: shutdown 2024-03-25T19:35:27.168 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:33.974 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:34.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.096+0000 7f2c73a63640 10 monclient: get_monmap_and_config 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.096+0000 7f2c73a63640 10 monclient: build_initial_monmap 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.096+0000 7f2c73a63640 10 monclient: monmap: 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:34.096925+0000 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:34.096925+0000 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:34.096 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.097 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:34.097 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:34.097 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:34.097 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.097+0000 7f2c73a63640 1 Processor -- start 2024-03-25T19:35:34.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.097+0000 7f2c73a63640 1 -- start start 2024-03-25T19:35:34.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.097+0000 7f2c73a63640 10 monclient: init 2024-03-25T19:35:34.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.097+0000 7f2c73a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:34.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:34.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c142520 con 0x7f2c6c13f710 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c0b28a0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c0b2a30 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:34.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.098+0000 7f2c73a63640 10 monclient(hunting): authenticate will time out at 5649.384147s 2024-03-25T19:35:34.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.110+0000 7f2c717ff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2c6c0a98f0 0x7f2c6c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34346/0 (socket says 172.21.15.47:34346) 2024-03-25T19:35:34.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.110+0000 7f2c717ff640 1 -- 172.21.15.47:0/1488645763 learned_addr learned my addr 172.21.15.47:0/1488645763 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:34.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.126+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276568896 0 0) 0x7f2c6c0b2a30 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.126+0000 7f2c6bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.126+0000 7f2c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.126+0000 7f2c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.126+0000 7f2c6bfff640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:35:34.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.126+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c54003520 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.134+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930548760 0 0) 0x7f2c6c0b28a0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.134+0000 7f2c6bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.134+0000 7f2c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.134+0000 7f2c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.134+0000 7f2c6bfff640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:35:34.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.134+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c6c0b2a30 con 0x7f2c6c1430c0 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980919146 0 0) 0x7f2c54003520 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 >> v1:172.21.15.142:6789/0 conn(0x7f2c6c1430c0 legacy=0x7f2c6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 >> v1:172.21.15.47:6789/0 conn(0x7f2c6c13f710 legacy=0x7f2c6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 1 monclient: found mon.noname-b 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c0b44c0 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: _finish_auth 0 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:34.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2c64001690 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c6bfff640 10 monclient: dump: 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c73a63640 5 monclient: authenticate success, global_id 4253 2024-03-25T19:35:34.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.140+0000 7f2c73a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:34.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c640017f0 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c6bfff640 1 -- 172.21.15.47:0/1488645763 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2c64001c00 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c6bfff640 10 monclient: dump: 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c73a63640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:34.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.162+0000 7f2c73a63640 4 set_mon_vals no callback set 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 10 monclient: shutdown 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 1 -- 172.21.15.47:0/1488645763 >> v1:172.21.15.47:6790/0 conn(0x7f2c6c0a98f0 legacy=0x7f2c6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 1 -- 172.21.15.47:0/1488645763 shutdown_connections 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 1 -- 172.21.15.47:0/1488645763 >> 172.21.15.47:0/1488645763 conn(0x7f2c6c0b7ea0 msgr2=0x7f2c6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 1 -- 172.21.15.47:0/1488645763 shutdown_connections 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 1 -- 172.21.15.47:0/1488645763 wait complete. 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 10 monclient: build_initial_monmap 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.163+0000 7f2c73a63640 10 monclient: monmap: 2024-03-25T19:35:34.165 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 librados: starting msgr at 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 librados: starting objecter 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 Processor -- start 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 -- start start 2024-03-25T19:35:34.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 librados: setting wanted keys 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 librados: calling monclient init 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient: init 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c0a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c1430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c1db7d0 con 0x7f2c6c13f710 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c0b1420 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c0b15b0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:34.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.164+0000 7f2c73a63640 10 monclient(hunting): authenticate will time out at 5649.450870s 2024-03-25T19:35:34.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.173+0000 7f2c717ff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2c6c0a98f0 0x7f2c6c1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:40162/0 (socket says 172.21.15.47:40162) 2024-03-25T19:35:34.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.173+0000 7f2c717ff640 1 -- 172.21.15.47:0/159402415 learned_addr learned my addr 172.21.15.47:0/159402415 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:34.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.176+0000 7f2c69ffb640 10 client.?.objecter ms_handle_connect 0x7f2c6c0a98f0 2024-03-25T19:35:34.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.176+0000 7f2c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:34.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.179+0000 7f2c69ffb640 10 client.?.objecter ms_handle_connect 0x7f2c6c13f710 2024-03-25T19:35:34.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.179+0000 7f2c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:34.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.183+0000 7f2c69ffb640 10 client.?.objecter ms_handle_connect 0x7f2c6c1430c0 2024-03-25T19:35:34.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.183+0000 7f2c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112329567 0 0) 0x7f2c6c0b1420 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c58004eb0 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947208614 0 0) 0x7f2c6c1db7d0 con 0x7f2c6c13f710 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:35:34.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.186+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c6c0b1420 con 0x7f2c6c13f710 2024-03-25T19:35:34.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.192+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366808636 0 0) 0x7f2c6c0b15b0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.192+0000 7f2c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.192+0000 7f2c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.192+0000 7f2c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.192+0000 7f2c69ffb640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:35:34.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.192+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c6c1db7d0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.201+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847245202 0 0) 0x7f2c58004eb0 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.201+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c6c0b15b0 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476481020 0 0) 0x7f2c6c1db7d0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c58004eb0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2c5c001bd0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:34.203 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 816302790 0 0) 0x7f2c6c0b1420 con 0x7f2c6c13f710 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c6c1db7d0 con 0x7f2c6c13f710 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2c600018b0 con 0x7f2c6c13f710 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.203+0000 7f2c69ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.204 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.204+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2c64001c60 con 0x7f2c6c0a98f0 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.204+0000 7f2c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.204+0000 7f2c69ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.204+0000 7f2c69ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:34.206 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3712339371 0 0) 0x7f2c58004eb0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 >> v1:172.21.15.142:6789/0 conn(0x7f2c6c0a98f0 legacy=0x7f2c6c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 >> v1:172.21.15.47:6789/0 conn(0x7f2c6c13f710 legacy=0x7f2c6c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 1 monclient: found mon.c 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c0b1740 con 0x7f2c6c1430c0 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:34.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 5 monclient: authenticate success, global_id 4256 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 10 monclient: _renew_subs 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6c1db1a0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 10 monclient: _renew_subs 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c0b08f0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.214+0000 7f2c73a63640 1 librados: init done 2024-03-25T19:35:34.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.215+0000 7f2c73a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.215+0000 7f2c73a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.215+0000 7f2c73a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:34.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.215+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c0627b0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c5c001310 con 0x7f2c6c1430c0 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2c5c003970 con 0x7f2c6c1430c0 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c69ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c69ffb640 10 monclient: dump: 2024-03-25T19:35:34.522 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:34.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.522+0000 7f2c6b7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:34.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.523+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f2c5c019940 con 0x7f2c6c1430c0 2024-03-25T19:35:34.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.523+0000 7f2c717ff640 10 monclient: get_auth_request con 0x7f2c5806e0d0 auth_method 0 2024-03-25T19:35:34.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.524+0000 7f2c69ffb640 10 client.4256.objecter ms_handle_connect 0x7f2c5806e0d0 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.529+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f2c5c053690 con 0x7f2c6c1430c0 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.529+0000 7f2c69ffb640 10 client.4256.objecter ms_dispatch 0x7f2c6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.529+0000 7f2c69ffb640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.529+0000 7f2c69ffb640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.529+0000 7f2c69ffb640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.529+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c5c04efd0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.530+0000 7f2c69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.530+0000 7f2c69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:34.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.742+0000 7f2c73a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.742+0000 7f2c73a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.742+0000 7f2c73a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:35:34.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.742+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c6c0abae0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.805+0000 7f2c69ffb640 1 -- 172.21.15.47:0/159402415 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f2c5c0510d0 con 0x7f2c6c1430c0 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.805+0000 7f2c69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.805+0000 7f2c69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:34.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.807+0000 7f2c73a63640 10 librados: watch_flush enter 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 10 librados: watch_flush exit 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 10 client.4256.objecter successfully canceled tick 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 >> v1:172.21.15.142:6800/1025302268 conn(0x7f2c5806e0d0 legacy=0x7f2c58070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 10 monclient: shutdown 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 >> v1:172.21.15.47:6790/0 conn(0x7f2c6c1430c0 legacy=0x7f2c6c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 shutdown_connections 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 >> 172.21.15.47:0/159402415 conn(0x7f2c6c0b7ea0 msgr2=0x7f2c6c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 shutdown_connections 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 -- 172.21.15.47:0/159402415 wait complete. 2024-03-25T19:35:34.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:34.808+0000 7f2c73a63640 1 librados: shutdown 2024-03-25T19:35:34.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:37.221 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:14.457 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.278 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:14.514 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.314 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:14.504 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.380 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:14.570 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.431 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:14.666 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.495 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:14.731 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.519 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:14.755 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.604 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:14.794 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.732 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:14.967 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:41.625 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:41.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.741+0000 7f243b2ae640 10 monclient: get_monmap_and_config 2024-03-25T19:35:41.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.741+0000 7f243b2ae640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.741+0000 7f243b2ae640 10 monclient: monmap: 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:41.741896+0000 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:41.741896+0000 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:41.741 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:41.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.741+0000 7f243b2ae640 1 Processor -- start 2024-03-25T19:35:41.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 1 -- start start 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient: init 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient(hunting): picked mon.noname-c con 0x7f24340a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient(hunting): picked mon.noname-a con 0x7f243413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient(hunting): picked mon.noname-b con 0x7f24341430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2434142520 con 0x7f243413f710 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24340b28a0 con 0x7f24340a98f0 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24340b2a30 con 0x7f24341430c0 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.742+0000 7f243b2ae640 10 monclient(hunting): authenticate will time out at 5657.028806s 2024-03-25T19:35:41.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.748+0000 7f2438849640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f243413f710 0x7f24341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55196/0 (socket says 172.21.15.47:55196) 2024-03-25T19:35:41.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.748+0000 7f2438849640 1 -- 172.21.15.47:0/537410255 learned_addr learned my addr 172.21.15.47:0/537410255 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:41.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.754+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074394841 0 0) 0x7f2434142520 con 0x7f243413f710 2024-03-25T19:35:41.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.754+0000 7f242bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.754+0000 7f242bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.754+0000 7f242bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.754+0000 7f242bfff640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:35:41.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.754+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2420003520 con 0x7f243413f710 2024-03-25T19:35:41.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.771+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391844143 0 0) 0x7f24340b2a30 con 0x7f24341430c0 2024-03-25T19:35:41.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.771+0000 7f242bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.771+0000 7f242bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.771+0000 7f242bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.771+0000 7f242bfff640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:35:41.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.771+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2434142520 con 0x7f24341430c0 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.775+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943588112 0 0) 0x7f24340b28a0 con 0x7f24340a98f0 2024-03-25T19:35:41.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.775+0000 7f242bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.775+0000 7f242bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.775+0000 7f242bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.775+0000 7f242bfff640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:35:41.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.775+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24340b2a30 con 0x7f24340a98f0 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917123217 0 0) 0x7f2420003520 con 0x7f243413f710 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 >> v1:172.21.15.47:6790/0 conn(0x7f24341430c0 legacy=0x7f243414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 >> v1:172.21.15.142:6789/0 conn(0x7f24340a98f0 legacy=0x7f24340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 1 monclient: found mon.noname-a 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:35:41.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24340b44c0 con 0x7f243413f710 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2424001690 con 0x7f243413f710 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f242bfff640 10 monclient: dump: 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:41.785 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:41.786 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:41.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f243b2ae640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:35:41.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.784+0000 7f243b2ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24240017f0 con 0x7f243413f710 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f242bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f242bfff640 1 -- 172.21.15.47:0/537410255 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2424001c00 con 0x7f243413f710 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 4 set_mon_vals no callback set 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f242bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f242bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:41.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f242bfff640 10 monclient: dump: 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:41.795 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:41.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 10 monclient: shutdown 2024-03-25T19:35:41.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:41.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 1 -- 172.21.15.47:0/537410255 >> v1:172.21.15.47:6789/0 conn(0x7f243413f710 legacy=0x7f24341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 1 -- 172.21.15.47:0/537410255 shutdown_connections 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 1 -- 172.21.15.47:0/537410255 >> 172.21.15.47:0/537410255 conn(0x7f24340b7ea0 msgr2=0x7f24340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 1 -- 172.21.15.47:0/537410255 shutdown_connections 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.795+0000 7f243b2ae640 1 -- 172.21.15.47:0/537410255 wait complete. 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.796+0000 7f243b2ae640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.796+0000 7f243b2ae640 10 monclient: monmap: 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:41.798 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.796+0000 7f243b2ae640 1 librados: starting msgr at 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.796+0000 7f243b2ae640 1 librados: starting objecter 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 1 Processor -- start 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 1 -- start start 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 1 librados: setting wanted keys 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 1 librados: calling monclient init 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 10 monclient: init 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:41.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 10 monclient(hunting): picked mon.noname-b con 0x7f24340a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 10 monclient(hunting): picked mon.noname-a con 0x7f243413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 10 monclient(hunting): picked mon.noname-c con 0x7f24341430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.797+0000 7f243b2ae640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24341d7d80 con 0x7f243413f710 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.798+0000 7f243b2ae640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24341d7f10 con 0x7f24340a98f0 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.798+0000 7f243b2ae640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24341d80a0 con 0x7f24341430c0 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.798+0000 7f243b2ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.798+0000 7f243b2ae640 10 monclient(hunting): authenticate will time out at 5657.083993s 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.801+0000 7f243904a640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f24340a98f0 0x7f24341d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:37138/0 (socket says 172.21.15.47:37138) 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.801+0000 7f243904a640 1 -- 172.21.15.47:0/1317210640 learned_addr learned my addr 172.21.15.47:0/1317210640 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.802+0000 7f2429ffb640 10 client.?.objecter ms_handle_connect 0x7f24340a98f0 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.802+0000 7f2429ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.811+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357281813 0 0) 0x7f24341d7f10 con 0x7f24340a98f0 2024-03-25T19:35:41.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.811+0000 7f2429ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.811+0000 7f2429ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.811+0000 7f2429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.811+0000 7f2429ffb640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:35:41.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.811+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2418004eb0 con 0x7f24340a98f0 2024-03-25T19:35:41.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.824+0000 7f2429ffb640 10 client.?.objecter ms_handle_connect 0x7f24341430c0 2024-03-25T19:35:41.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.824+0000 7f2429ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.827+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3554095955 0 0) 0x7f2418004eb0 con 0x7f24340a98f0 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24341d7f10 con 0x7f24340a98f0 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2430001a10 con 0x7f24340a98f0 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 10 client.?.objecter ms_handle_connect 0x7f243413f710 2024-03-25T19:35:41.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.828+0000 7f2429ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.849+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912498062 0 0) 0x7f24341d7d80 con 0x7f243413f710 2024-03-25T19:35:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.849+0000 7f2429ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.849+0000 7f2429ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.849+0000 7f2429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.849+0000 7f2429ffb640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:35:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.849+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2418004eb0 con 0x7f243413f710 2024-03-25T19:35:41.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3051264970 0 0) 0x7f24341d7f10 con 0x7f24340a98f0 2024-03-25T19:35:41.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 >> v1:172.21.15.47:6790/0 conn(0x7f24341430c0 legacy=0x7f2434135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 >> v1:172.21.15.47:6789/0 conn(0x7f243413f710 legacy=0x7f2434135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 1 monclient: found mon.b 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24340b7200 con 0x7f24340a98f0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f2429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f243b2ae640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f243b2ae640 10 monclient: _renew_subs 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f243b2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.852+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24340b7450 con 0x7f24340a98f0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.853+0000 7f243b2ae640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.853+0000 7f243b2ae640 10 monclient: _renew_subs 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.853+0000 7f243b2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.853+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24341e3230 con 0x7f24340a98f0 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.853+0000 7f243b2ae640 1 librados: init done 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.854+0000 7f243b2ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.854+0000 7f243b2ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.854+0000 7f243b2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:41.854+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24341d84a0 con 0x7f24340a98f0 2024-03-25T19:35:42.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2430001cf0 con 0x7f24340a98f0 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f2429ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2430003a60 con 0x7f24340a98f0 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f2429ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f2429ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f2429ffb640 10 monclient: dump: 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:42.131 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:42.132 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:42.132 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:42.132 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:42.132 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:42.132 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:42.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.131+0000 7f242b7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:42.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.132+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f2430003c80 con 0x7f24340a98f0 2024-03-25T19:35:42.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.132+0000 7f2438849640 10 monclient: get_auth_request con 0x7f241806dd90 auth_method 0 2024-03-25T19:35:42.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.132+0000 7f2429ffb640 10 client.4267.objecter ms_handle_connect 0x7f241806dd90 2024-03-25T19:35:42.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.136+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f2430054880 con 0x7f24340a98f0 2024-03-25T19:35:42.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.136+0000 7f2429ffb640 10 client.4267.objecter ms_dispatch 0x7f2434000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:42.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.136+0000 7f2429ffb640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:42.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.136+0000 7f2429ffb640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:42.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.136+0000 7f2429ffb640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.138+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2430050390 con 0x7f24340a98f0 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.138+0000 7f2429ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:42.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.138+0000 7f2429ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:42.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.390+0000 7f243b2ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.390+0000 7f243b2ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.390+0000 7f243b2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:42.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.390+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24340627b0 con 0x7f24340a98f0 2024-03-25T19:35:42.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.417+0000 7f2429ffb640 1 -- 172.21.15.47:0/1317210640 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f24300522c0 con 0x7f24340a98f0 2024-03-25T19:35:42.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.417+0000 7f2429ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.417+0000 7f2429ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:42.417 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:42.417 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:42.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 10 librados: watch_flush enter 2024-03-25T19:35:42.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 10 librados: watch_flush exit 2024-03-25T19:35:42.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 10 client.4267.objecter successfully canceled tick 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 >> v1:172.21.15.142:6800/1025302268 conn(0x7f241806dd90 legacy=0x7f2418070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 10 monclient: shutdown 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.419+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 >> v1:172.21.15.142:6789/0 conn(0x7f24340a98f0 legacy=0x7f24341d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.420+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 shutdown_connections 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.420+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 >> 172.21.15.47:0/1317210640 conn(0x7f24340b7ea0 msgr2=0x7f2434074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.420+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 shutdown_connections 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.420+0000 7f243b2ae640 1 -- 172.21.15.47:0/1317210640 wait complete. 2024-03-25T19:35:42.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:42.420+0000 7f243b2ae640 1 librados: shutdown 2024-03-25T19:35:42.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:49.236 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.351+0000 7fc5453a0640 10 monclient: get_monmap_and_config 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.351+0000 7fc5453a0640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.351+0000 7fc5453a0640 10 monclient: monmap: 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:49.352540+0000 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:49.352540+0000 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.351 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:49.352 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:49.352 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:49.352 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:49.352 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.352+0000 7fc5453a0640 1 Processor -- start 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.352+0000 7fc5453a0640 1 -- start start 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.352+0000 7fc5453a0640 10 monclient: init 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient(hunting): picked mon.noname-b con 0x7fc5400b48e0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient(hunting): picked mon.noname-c con 0x7fc54013fc60 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:49.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient(hunting): picked mon.noname-a con 0x7fc5400b3f90 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:49.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5400a9d10 con 0x7fc5400b3f90 2024-03-25T19:35:49.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5400ab000 con 0x7fc54013fc60 2024-03-25T19:35:49.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5400a9680 con 0x7fc5400b48e0 2024-03-25T19:35:49.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.353+0000 7fc5453a0640 10 monclient(hunting): authenticate will time out at 5664.639502s 2024-03-25T19:35:49.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.362+0000 7fc53f7fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc54013fc60 0x7fc540145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:48002/0 (socket says 172.21.15.47:48002) 2024-03-25T19:35:49.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.362+0000 7fc53f7fe640 1 -- 172.21.15.47:0/3653978034 learned_addr learned my addr 172.21.15.47:0/3653978034 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:49.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.371+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885948512 0 0) 0x7fc5400a9680 con 0x7fc5400b48e0 2024-03-25T19:35:49.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.371+0000 7fc53effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.371+0000 7fc53effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.371+0000 7fc53effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.371+0000 7fc53effd640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:35:49.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.371+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528003520 con 0x7fc5400b48e0 2024-03-25T19:35:49.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.375+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841272837 0 0) 0x7fc5400a9d10 con 0x7fc5400b3f90 2024-03-25T19:35:49.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.375+0000 7fc53effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.375+0000 7fc53effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.375+0000 7fc53effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.375+0000 7fc53effd640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:35:49.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.375+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5400a9680 con 0x7fc5400b3f90 2024-03-25T19:35:49.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.382+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991318107 0 0) 0x7fc5400ab000 con 0x7fc54013fc60 2024-03-25T19:35:49.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.382+0000 7fc53effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.382+0000 7fc53effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.382+0000 7fc53effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.382+0000 7fc53effd640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:35:49.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.382+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5400a9d10 con 0x7fc54013fc60 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809819086 0 0) 0x7fc5400a9680 con 0x7fc5400b3f90 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 >> v1:172.21.15.47:6790/0 conn(0x7fc5400b48e0 legacy=0x7fc54013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 >> v1:172.21.15.142:6789/0 conn(0x7fc54013fc60 legacy=0x7fc540145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 1 monclient: found mon.noname-a 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5400a9810 con 0x7fc5400b3f90 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc5380019f0 con 0x7fc5400b3f90 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc53effd640 10 monclient: dump: 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc5453a0640 5 monclient: authenticate success, global_id 4278 2024-03-25T19:35:49.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.387+0000 7fc5453a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.392+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc538001b50 con 0x7fc5400b3f90 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.392+0000 7fc53effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.392+0000 7fc53effd640 1 -- 172.21.15.47:0/3653978034 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc538001f60 con 0x7fc5400b3f90 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.392+0000 7fc53effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.392+0000 7fc53effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.392+0000 7fc53effd640 10 monclient: dump: 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:49.392 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 4 set_mon_vals no callback set 2024-03-25T19:35:49.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 10 monclient: shutdown 2024-03-25T19:35:49.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 1 -- 172.21.15.47:0/3653978034 >> v1:172.21.15.47:6789/0 conn(0x7fc5400b3f90 legacy=0x7fc54014dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 1 -- 172.21.15.47:0/3653978034 shutdown_connections 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 1 -- 172.21.15.47:0/3653978034 >> 172.21.15.47:0/3653978034 conn(0x7fc5400615c0 msgr2=0x7fc5400619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 1 -- 172.21.15.47:0/3653978034 shutdown_connections 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.393+0000 7fc5453a0640 1 -- 172.21.15.47:0/3653978034 wait complete. 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.394+0000 7fc5453a0640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.394+0000 7fc5453a0640 10 monclient: monmap: 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:49.395 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 1 librados: starting msgr at 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 1 librados: starting objecter 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 1 Processor -- start 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 1 -- start start 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 1 librados: setting wanted keys 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 1 librados: calling monclient init 2024-03-25T19:35:49.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.395+0000 7fc5453a0640 10 monclient: init 2024-03-25T19:35:49.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:49.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient(hunting): picked mon.noname-b con 0x7fc5400b3f90 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:49.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient(hunting): picked mon.noname-a con 0x7fc5400b48e0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:49.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient(hunting): picked mon.noname-c con 0x7fc54013fc60 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:49.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54013aa40 con 0x7fc5400b48e0 2024-03-25T19:35:49.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5400adcd0 con 0x7fc5400b3f90 2024-03-25T19:35:49.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5400acdb0 con 0x7fc54013fc60 2024-03-25T19:35:49.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.396+0000 7fc5453a0640 10 monclient(hunting): authenticate will time out at 5664.682513s 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.410+0000 7fc544b9f640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc54013fc60 0x7fc5401e2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55994/0 (socket says 172.21.15.47:55994) 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.410+0000 7fc544b9f640 1 -- 172.21.15.47:0/2889917037 learned_addr learned my addr 172.21.15.47:0/2889917037 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:49.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.411+0000 7fc53cff9640 10 client.?.objecter ms_handle_connect 0x7fc54013fc60 2024-03-25T19:35:49.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.412+0000 7fc53cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.416+0000 7fc53cff9640 10 client.?.objecter ms_handle_connect 0x7fc5400b3f90 2024-03-25T19:35:49.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.416+0000 7fc53cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.418+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307606725 0 0) 0x7fc5400acdb0 con 0x7fc54013fc60 2024-03-25T19:35:49.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.418+0000 7fc53cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.418+0000 7fc53cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.418+0000 7fc53cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.418+0000 7fc53cff9640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.418+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c004eb0 con 0x7fc54013fc60 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.419+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080405602 0 0) 0x7fc5400adcd0 con 0x7fc5400b3f90 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.419+0000 7fc53cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.419+0000 7fc53cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.419+0000 7fc53cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.419+0000 7fc53cff9640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:35:49.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.419+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5400acdb0 con 0x7fc5400b3f90 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964354378 0 0) 0x7fc5400acdb0 con 0x7fc5400b3f90 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5400adcd0 con 0x7fc5400b3f90 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc530001950 con 0x7fc5400b3f90 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 10 monclient(hunting): dump: 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.430+0000 7fc53cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.433+0000 7fc53cff9640 10 client.?.objecter ms_handle_connect 0x7fc5400b48e0 2024-03-25T19:35:49.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.433+0000 7fc53cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1218468844 0 0) 0x7fc5400adcd0 con 0x7fc5400b3f90 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 >> v1:172.21.15.47:6790/0 conn(0x7fc54013fc60 legacy=0x7fc5401e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 >> v1:172.21.15.47:6789/0 conn(0x7fc5400b48e0 legacy=0x7fc5400b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 1 monclient: found mon.b 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5400ad500 con 0x7fc5400b3f90 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc53cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 10 monclient: _renew_subs 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5400ae1f0 con 0x7fc5400b3f90 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 10 monclient: _renew_subs 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5400ad990 con 0x7fc5400b3f90 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.447+0000 7fc5453a0640 1 librados: init done 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.448+0000 7fc5453a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.448+0000 7fc5453a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.448+0000 7fc5453a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.448+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5400ade60 con 0x7fc5400b3f90 2024-03-25T19:35:49.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc530001d60 con 0x7fc5400b3f90 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc5300037b0 con 0x7fc5400b3f90 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53cff9640 10 monclient: dump: 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:49.737 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:49.738 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:49.738 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:49.738 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:49.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.737+0000 7fc53e7fc640 4 set_mon_vals no callback set 2024-03-25T19:35:49.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.738+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fc530003990 con 0x7fc5400b3f90 2024-03-25T19:35:49.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.738+0000 7fc53f7fe640 10 monclient: get_auth_request con 0x7fc52c06dcb0 auth_method 0 2024-03-25T19:35:49.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.738+0000 7fc53cff9640 10 client.4273.objecter ms_handle_connect 0x7fc52c06dcb0 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.742+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fc5300535b0 con 0x7fc5400b3f90 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.742+0000 7fc53cff9640 10 client.4273.objecter ms_dispatch 0x7fc540000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.742+0000 7fc53cff9640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.742+0000 7fc53cff9640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.742+0000 7fc53cff9640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.743+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc53004eef0 con 0x7fc5400b3f90 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.743+0000 7fc53cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:49.743+0000 7fc53cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:50.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.012+0000 7fc5453a0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:50.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.012+0000 7fc5453a0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:50.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.012+0000 7fc5453a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:50.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.012+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5400aaef0 con 0x7fc5400b3f90 2024-03-25T19:35:50.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.041+0000 7fc53cff9640 1 -- 172.21.15.47:0/2889917037 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fc530050ff0 con 0x7fc5400b3f90 2024-03-25T19:35:50.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.041+0000 7fc53cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:50.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.041+0000 7fc53cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:50.041 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:50.041 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 10 librados: watch_flush enter 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 10 librados: watch_flush exit 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 10 client.4273.objecter successfully canceled tick 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 >> v1:172.21.15.142:6800/1025302268 conn(0x7fc52c06dcb0 legacy=0x7fc52c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 10 monclient: shutdown 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 >> v1:172.21.15.142:6789/0 conn(0x7fc5400b3f90 legacy=0x7fc5401d7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 shutdown_connections 2024-03-25T19:35:50.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 >> 172.21.15.47:0/2889917037 conn(0x7fc5400615c0 msgr2=0x7fc5400b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 shutdown_connections 2024-03-25T19:35:50.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 -- 172.21.15.47:0/2889917037 wait complete. 2024-03-25T19:35:50.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:50.044+0000 7fc5453a0640 1 librados: shutdown 2024-03-25T19:35:50.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:56.861 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.979+0000 7f11cde3e640 10 monclient: get_monmap_and_config 2024-03-25T19:35:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.979+0000 7f11cde3e640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.979+0000 7f11cde3e640 10 monclient: monmap: 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:35:56.980984+0000 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:35:56.980984+0000 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:35:56.980 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.980+0000 7f11cde3e640 1 Processor -- start 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.980+0000 7f11cde3e640 1 -- start start 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.980+0000 7f11cde3e640 10 monclient: init 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient(hunting): picked mon.noname-a con 0x7f11c80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient(hunting): picked mon.noname-c con 0x7f11c813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient(hunting): picked mon.noname-b con 0x7f11c81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c8142520 con 0x7f11c80a98f0 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c80b28a0 con 0x7f11c813f710 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c80b2a30 con 0x7f11c81430c0 2024-03-25T19:35:56.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.981+0000 7f11cde3e640 10 monclient(hunting): authenticate will time out at 5672.268176s 2024-03-25T19:35:56.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.988+0000 7f11c7fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f11c81430c0 0x7f11c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60542/0 (socket says 172.21.15.47:60542) 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:56.988+0000 7f11c7fff640 1 -- 172.21.15.47:0/2030534341 learned_addr learned my addr 172.21.15.47:0/2030534341 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:57.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.006+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916635188 0 0) 0x7f11c8142520 con 0x7f11c80a98f0 2024-03-25T19:35:57.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.007+0000 7f11c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:57.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.007+0000 7f11c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:57.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.007+0000 7f11c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:57.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.007+0000 7f11c67fc640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:35:57.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.007+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b0003520 con 0x7f11c80a98f0 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418114916 0 0) 0x7f11b0003520 con 0x7f11c80a98f0 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 >> v1:172.21.15.47:6790/0 conn(0x7f11c81430c0 legacy=0x7f11c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 >> v1:172.21.15.142:6789/0 conn(0x7f11c813f710 legacy=0x7f11c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 1 monclient: found mon.noname-a 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:57.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80b44c0 con 0x7f11c80a98f0 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f11b8001690 con 0x7f11c80a98f0 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11cde3e640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11cde3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.013+0000 7f11c67fc640 10 monclient: dump: 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:57.015 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:57.016 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:57.016 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:57.016 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:57.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.014+0000 7f11cde3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:57.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b80017f0 con 0x7f11c80a98f0 2024-03-25T19:35:57.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11c67fc640 1 -- 172.21.15.47:0/2030534341 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f11b8001c00 con 0x7f11c80a98f0 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11c67fc640 10 monclient: dump: 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:57.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11cde3e640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:57.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.018+0000 7f11cde3e640 4 set_mon_vals no callback set 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 10 monclient: shutdown 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 1 -- 172.21.15.47:0/2030534341 >> v1:172.21.15.47:6789/0 conn(0x7f11c80a98f0 legacy=0x7f11c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 1 -- 172.21.15.47:0/2030534341 shutdown_connections 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 1 -- 172.21.15.47:0/2030534341 >> 172.21.15.47:0/2030534341 conn(0x7f11c80b7ea0 msgr2=0x7f11c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 1 -- 172.21.15.47:0/2030534341 shutdown_connections 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.019+0000 7f11cde3e640 1 -- 172.21.15.47:0/2030534341 wait complete. 2024-03-25T19:35:57.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.020+0000 7f11cde3e640 10 monclient: build_initial_monmap 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.020+0000 7f11cde3e640 10 monclient: monmap: 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.020+0000 7f11cde3e640 1 librados: starting msgr at 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.020+0000 7f11cde3e640 1 librados: starting objecter 2024-03-25T19:35:57.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.020+0000 7f11cde3e640 1 Processor -- start 2024-03-25T19:35:57.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 1 -- start start 2024-03-25T19:35:57.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 1 librados: setting wanted keys 2024-03-25T19:35:57.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 1 librados: calling monclient init 2024-03-25T19:35:57.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 10 monclient: init 2024-03-25T19:35:57.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 10 monclient(hunting): picked mon.noname-c con 0x7f11c80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 10 monclient(hunting): picked mon.noname-b con 0x7f11c813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.021+0000 7f11cde3e640 10 monclient(hunting): picked mon.noname-a con 0x7f11c81430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.022+0000 7f11cde3e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c8138c70 con 0x7f11c81430c0 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.022+0000 7f11cde3e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c81367d0 con 0x7f11c813f710 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.022+0000 7f11cde3e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c8136960 con 0x7f11c80a98f0 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.022+0000 7f11cde3e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:57.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.022+0000 7f11cde3e640 10 monclient(hunting): authenticate will time out at 5672.309018s 2024-03-25T19:35:57.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.026+0000 7f11c7fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f11c81430c0 0x7f11c81e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53226/0 (socket says 172.21.15.47:53226) 2024-03-25T19:35:57.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.026+0000 7f11c7fff640 1 -- 172.21.15.47:0/3586613836 learned_addr learned my addr 172.21.15.47:0/3586613836 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:35:57.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.026+0000 7f11affff640 10 client.?.objecter ms_handle_connect 0x7f11c81430c0 2024-03-25T19:35:57.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.026+0000 7f11affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:57.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.033+0000 7f11affff640 10 client.?.objecter ms_handle_connect 0x7f11c813f710 2024-03-25T19:35:57.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.033+0000 7f11affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:57.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.042+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307859925 0 0) 0x7f11c81367d0 con 0x7f11c813f710 2024-03-25T19:35:57.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.042+0000 7f11affff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:57.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.042+0000 7f11affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:57.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.042+0000 7f11affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:57.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.042+0000 7f11affff640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:35:57.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.042+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b4004eb0 con 0x7f11c813f710 2024-03-25T19:35:57.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.054+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280846045 0 0) 0x7f11c8138c70 con 0x7f11c81430c0 2024-03-25T19:35:57.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.054+0000 7f11affff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:57.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.054+0000 7f11affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:57.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.054+0000 7f11affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:57.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.054+0000 7f11affff640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:35:57.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.054+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c81367d0 con 0x7f11c81430c0 2024-03-25T19:35:57.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.059+0000 7f11affff640 10 client.?.objecter ms_handle_connect 0x7f11c80a98f0 2024-03-25T19:35:57.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.059+0000 7f11affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:57.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355765174 0 0) 0x7f11b4004eb0 con 0x7f11c813f710 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11c8138c70 con 0x7f11c813f710 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f11bc001930 con 0x7f11c813f710 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 10 monclient(hunting): dump: 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:57.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.063+0000 7f11affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1773879232 0 0) 0x7f11c8138c70 con 0x7f11c813f710 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 >> v1:172.21.15.47:6790/0 conn(0x7f11c80a98f0 legacy=0x7f11c8137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 >> v1:172.21.15.47:6789/0 conn(0x7f11c81430c0 legacy=0x7f11c81e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 1 monclient: found mon.b 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c8139630 con 0x7f11c813f710 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 10 monclient: _finish_auth 0 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:57.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 10 monclient: _renew_subs 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c8139880 con 0x7f11c813f710 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 10 monclient: _renew_subs 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c81337f0 con 0x7f11c813f710 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.070+0000 7f11cde3e640 1 librados: init done 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.071+0000 7f11cde3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.071+0000 7f11cde3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.071+0000 7f11cde3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:57.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.071+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c80627b0 con 0x7f11c813f710 2024-03-25T19:35:57.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11bc003610 con 0x7f11c813f710 2024-03-25T19:35:57.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f11bc003a20 con 0x7f11c813f710 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11affff640 10 monclient: dump: 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:35:57.347 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.345+0000 7f11c5ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.346+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f11bc003c40 con 0x7f11c813f710 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.347+0000 7f11c77fe640 10 monclient: get_auth_request con 0x7f11b406dd10 auth_method 0 2024-03-25T19:35:57.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.347+0000 7f11affff640 10 client.4279.objecter ms_handle_connect 0x7f11b406dd10 2024-03-25T19:35:57.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.351+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f11bc0535b0 con 0x7f11c813f710 2024-03-25T19:35:57.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.351+0000 7f11affff640 10 client.4279.objecter ms_dispatch 0x7f11c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:57.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.351+0000 7f11affff640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:57.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.351+0000 7f11affff640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:57.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.351+0000 7f11affff640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T19:35:57.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.352+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11bc04f0c0 con 0x7f11c813f710 2024-03-25T19:35:57.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.353+0000 7f11affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.353+0000 7f11affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:57.415 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:34.650 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.418 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:34.653 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.424 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:34.614 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.429 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:34.619 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.472 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:34.662 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.490 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:34.725 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.502 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:34.737 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.510 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:34.746 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.559 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:34.794 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.569+0000 7f11cde3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.569+0000 7f11cde3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.569+0000 7f11cde3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.569+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11c8069f40 con 0x7f11c813f710 2024-03-25T19:35:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.678+0000 7f11affff640 1 -- 172.21.15.47:0/3586613836 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f11bc050ff0 con 0x7f11c813f710 2024-03-25T19:35:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.678+0000 7f11affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.678+0000 7f11affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:57.679 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:35:57.679 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 10 librados: watch_flush enter 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 10 librados: watch_flush exit 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 10 client.4279.objecter successfully canceled tick 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 >> v1:172.21.15.142:6800/1025302268 conn(0x7f11b406dd10 legacy=0x7f11b40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 10 monclient: shutdown 2024-03-25T19:35:57.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:57.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 >> v1:172.21.15.142:6789/0 conn(0x7f11c813f710 legacy=0x7f11c8132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 shutdown_connections 2024-03-25T19:35:57.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 >> 172.21.15.47:0/3586613836 conn(0x7f11c80b7ea0 msgr2=0x7f11c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:57.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 shutdown_connections 2024-03-25T19:35:57.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 -- 172.21.15.47:0/3586613836 wait complete. 2024-03-25T19:35:57.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:35:57.681+0000 7f11cde3e640 1 librados: shutdown 2024-03-25T19:35:57.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:04.498 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:04.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.611+0000 7ff2b33d0640 10 monclient: get_monmap_and_config 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.611+0000 7ff2b33d0640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.611+0000 7ff2b33d0640 10 monclient: monmap: 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:04.612433+0000 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:04.612433+0000 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.612+0000 7ff2b33d0640 1 Processor -- start 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.612+0000 7ff2b33d0640 1 -- start start 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.612+0000 7ff2b33d0640 10 monclient: init 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ac0b10d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ac0b4360 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ac0a9470 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac0a9ab0 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac0b2400 con 0x7ff2ac0a9470 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac0aa2a0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.613+0000 7ff2b33d0640 10 monclient(hunting): authenticate will time out at 5679.899312s 2024-03-25T19:36:04.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.625+0000 7ff2b096b640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff2ac0b4360 0x7ff2ac0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:50108/0 (socket says 172.21.15.47:50108) 2024-03-25T19:36:04.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.625+0000 7ff2b096b640 1 -- 172.21.15.47:0/2885843636 learned_addr learned my addr 172.21.15.47:0/2885843636 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:04.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.647+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974337101 0 0) 0x7ff2ac0a9ab0 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.647+0000 7ff2a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.647+0000 7ff2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.647+0000 7ff2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.647+0000 7ff2a3fff640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:36:04.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.647+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff298003520 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.651+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442185494 0 0) 0x7ff2ac0aa2a0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.651+0000 7ff2a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.651+0000 7ff2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.651+0000 7ff2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.651+0000 7ff2a3fff640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:36:04.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.651+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac0a9ab0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.653+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042988598 0 0) 0x7ff2ac0b2400 con 0x7ff2ac0a9470 2024-03-25T19:36:04.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.653+0000 7ff2a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.653+0000 7ff2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.653+0000 7ff2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.653+0000 7ff2a3fff640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:36:04.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.653+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac0aa2a0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815675332 0 0) 0x7ff298003520 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 >> v1:172.21.15.47:6790/0 conn(0x7ff2ac0b4360 legacy=0x7ff2ac0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 >> v1:172.21.15.142:6789/0 conn(0x7ff2ac0a9470 legacy=0x7ff2ac14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 1 monclient: found mon.noname-a 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0b14b0 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff2a8001690 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2a3fff640 10 monclient: dump: 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.661 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2b33d0640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:36:04.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.661+0000 7ff2b33d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:04.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2a80017f0 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2a3fff640 1 -- 172.21.15.47:0/2885843636 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff2a8001c00 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2b33d0640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2b33d0640 4 set_mon_vals no callback set 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.672+0000 7ff2a3fff640 10 monclient: dump: 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:04.672 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:04.673 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 10 monclient: shutdown 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 1 -- 172.21.15.47:0/2885843636 >> v1:172.21.15.47:6789/0 conn(0x7ff2ac0b10d0 legacy=0x7ff2ac0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 1 -- 172.21.15.47:0/2885843636 shutdown_connections 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 1 -- 172.21.15.47:0/2885843636 >> 172.21.15.47:0/2885843636 conn(0x7ff2ac0617a0 msgr2=0x7ff2ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 1 -- 172.21.15.47:0/2885843636 shutdown_connections 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.673+0000 7ff2b33d0640 1 -- 172.21.15.47:0/2885843636 wait complete. 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 10 monclient: monmap: 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 1 librados: starting msgr at 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 1 librados: starting objecter 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 1 Processor -- start 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 1 -- start start 2024-03-25T19:36:04.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 1 librados: setting wanted keys 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 1 librados: calling monclient init 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.674+0000 7ff2b33d0640 10 monclient: init 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ac0a9470 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ac0b10d0 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ac0b4360 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac0ac0d0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac1d5180 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac1d83d0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.675+0000 7ff2b33d0640 10 monclient(hunting): authenticate will time out at 5679.961658s 2024-03-25T19:36:04.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.685+0000 7ff2b116c640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff2ac0a9470 0x7ff2ac0ab9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:50146/0 (socket says 172.21.15.47:50146) 2024-03-25T19:36:04.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.686+0000 7ff2b116c640 1 -- 172.21.15.47:0/107774393 learned_addr learned my addr 172.21.15.47:0/107774393 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:04.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.686+0000 7ff2a1ffb640 10 client.?.objecter ms_handle_connect 0x7ff2ac0a9470 2024-03-25T19:36:04.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.686+0000 7ff2a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.693+0000 7ff2a1ffb640 10 client.?.objecter ms_handle_connect 0x7ff2ac0b10d0 2024-03-25T19:36:04.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.693+0000 7ff2a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.698+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843967617 0 0) 0x7ff2ac1d83d0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.698+0000 7ff2a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.698+0000 7ff2a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.698+0000 7ff2a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.698+0000 7ff2a1ffb640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:36:04.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.699+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff290004eb0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.701+0000 7ff2a1ffb640 10 client.?.objecter ms_handle_connect 0x7ff2ac0b4360 2024-03-25T19:36:04.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.701+0000 7ff2a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229897420 0 0) 0x7ff2ac0aa2a0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac1d83d0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148743804 0 0) 0x7ff2ac1d5180 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:36:04.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.705+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac0aa2a0 con 0x7ff2ac0b10d0 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.711+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1456897177 0 0) 0x7ff290004eb0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.712+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2ac1d5180 con 0x7ff2ac0a9470 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.712+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff2a8001bc0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.712+0000 7ff2a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.712+0000 7ff2a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.712+0000 7ff2a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:04.711 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:04.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.712+0000 7ff2a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:04.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.716+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119581420 0 0) 0x7ff2ac1d83d0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.716+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff290004eb0 con 0x7ff2ac0b4360 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.716+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff2a4001a50 con 0x7ff2ac0b4360 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.716+0000 7ff2a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.716+0000 7ff2a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.716+0000 7ff2a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:04.716 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:04.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2075150702 0 0) 0x7ff2ac1d5180 con 0x7ff2ac0a9470 2024-03-25T19:36:04.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 >> v1:172.21.15.142:6789/0 conn(0x7ff2ac0b10d0 legacy=0x7ff2ac1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 >> v1:172.21.15.47:6789/0 conn(0x7ff2ac0b4360 legacy=0x7ff2ac1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 1 monclient: found mon.c 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0ad9b0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 5 monclient: authenticate success, global_id 4280 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 10 monclient: _renew_subs 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac1d77a0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 10 monclient: _renew_subs 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:36:04.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac0ac9c0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.721+0000 7ff2b33d0640 1 librados: init done 2024-03-25T19:36:04.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.722+0000 7ff2b33d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.722+0000 7ff2b33d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.722+0000 7ff2b33d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:36:04.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.722+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac1d7590 con 0x7ff2ac0a9470 2024-03-25T19:36:04.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.728+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2a80011f0 con 0x7ff2ac0a9470 2024-03-25T19:36:04.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.728+0000 7ff2a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:04.728+0000 7ff2a37fe640 4 set_mon_vals no callback set 2024-03-25T19:36:05.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.041+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff2a8003b00 con 0x7ff2ac0a9470 2024-03-25T19:36:05.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.041+0000 7ff2a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:05.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.041+0000 7ff2a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:05.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.041+0000 7ff2a1ffb640 10 monclient: dump: 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:05.041 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:05.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.041+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ff2a8003ce0 con 0x7ff2ac0a9470 2024-03-25T19:36:05.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.042+0000 7ff2b096b640 10 monclient: get_auth_request con 0x7ff29006df70 auth_method 0 2024-03-25T19:36:05.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.042+0000 7ff2a1ffb640 10 client.4280.objecter ms_handle_connect 0x7ff29006df70 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7ff2a8054900 con 0x7ff2ac0a9470 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 10 client.4280.objecter ms_dispatch 0x7ff2ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2a8050240 con 0x7ff2ac0a9470 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:05.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.048+0000 7ff2a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.299+0000 7ff2b33d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.299+0000 7ff2b33d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.299+0000 7ff2b33d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.299+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2ac1d7930 con 0x7ff2ac0a9470 2024-03-25T19:36:05.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.417+0000 7ff2a1ffb640 1 -- 172.21.15.47:0/107774393 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7ff2a8052340 con 0x7ff2ac0a9470 2024-03-25T19:36:05.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.417+0000 7ff2a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.417+0000 7ff2a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:05.417 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:05.417 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 10 librados: watch_flush enter 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 10 librados: watch_flush exit 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 10 client.4280.objecter successfully canceled tick 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 >> v1:172.21.15.142:6800/1025302268 conn(0x7ff29006df70 legacy=0x7ff290070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 10 monclient: shutdown 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 >> v1:172.21.15.47:6790/0 conn(0x7ff2ac0a9470 legacy=0x7ff2ac0ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 shutdown_connections 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 >> 172.21.15.47:0/107774393 conn(0x7ff2ac0617a0 msgr2=0x7ff2ac145e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 shutdown_connections 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 -- 172.21.15.47:0/107774393 wait complete. 2024-03-25T19:36:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:05.420+0000 7ff2b33d0640 1 librados: shutdown 2024-03-25T19:36:05.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:12.238 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.356+0000 7f64bae95640 10 monclient: get_monmap_and_config 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.356+0000 7f64bae95640 10 monclient: build_initial_monmap 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.356+0000 7f64bae95640 10 monclient: monmap: 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:12.356800+0000 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:12.356800+0000 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:12.355 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:12.356 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.356+0000 7f64bae95640 1 Processor -- start 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.356+0000 7f64bae95640 1 -- start start 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.356+0000 7f64bae95640 10 monclient: init 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient(hunting): picked mon.noname-c con 0x7f64b40b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient(hunting): picked mon.noname-a con 0x7f64b40b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient(hunting): picked mon.noname-b con 0x7f64b414be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:12.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40af940 con 0x7f64b40b1730 2024-03-25T19:36:12.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40b0a70 con 0x7f64b40b4d60 2024-03-25T19:36:12.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40b0d50 con 0x7f64b414be90 2024-03-25T19:36:12.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:12.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.357+0000 7f64bae95640 10 monclient(hunting): authenticate will time out at 5687.643401s 2024-03-25T19:36:12.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.358+0000 7f64b9432640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f64b414be90 0x7f64b414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39044/0 (socket says 172.21.15.47:39044) 2024-03-25T19:36:12.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.358+0000 7f64b9432640 1 -- 172.21.15.47:0/1773340849 learned_addr learned my addr 172.21.15.47:0/1773340849 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:12.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.435+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045828636 0 0) 0x7f64b40af940 con 0x7f64b40b1730 2024-03-25T19:36:12.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.435+0000 7f64b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.435+0000 7f64b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.435+0000 7f64b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.435+0000 7f64b37fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:36:12.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.435+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4003520 con 0x7f64b40b1730 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037766261 0 0) 0x7f64a4003520 con 0x7f64b40b1730 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 >> v1:172.21.15.47:6790/0 conn(0x7f64b414be90 legacy=0x7f64b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 >> v1:172.21.15.142:6789/0 conn(0x7f64b40b4d60 legacy=0x7f64b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 1 monclient: found mon.noname-a 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:12.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40b1030 con 0x7f64b40b1730 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f64a0001690 con 0x7f64b40b1730 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64bae95640 5 monclient: authenticate success, global_id 4296 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64bae95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64b37fe640 10 monclient: dump: 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.444 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:12.445 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:12.445 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:12.445 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:12.445 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:12.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.443+0000 7f64bae95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:12.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a00017f0 con 0x7f64b40b1730 2024-03-25T19:36:12.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64b37fe640 1 -- 172.21.15.47:0/1773340849 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f64a0001c00 con 0x7f64b40b1730 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64b37fe640 10 monclient: dump: 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64bae95640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:12.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.448+0000 7f64bae95640 4 set_mon_vals no callback set 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 10 monclient: shutdown 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 1 -- 172.21.15.47:0/1773340849 >> v1:172.21.15.47:6789/0 conn(0x7f64b40b1730 legacy=0x7f64b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 1 -- 172.21.15.47:0/1773340849 shutdown_connections 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 1 -- 172.21.15.47:0/1773340849 >> 172.21.15.47:0/1773340849 conn(0x7f64b40617a0 msgr2=0x7f64b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 1 -- 172.21.15.47:0/1773340849 shutdown_connections 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.449+0000 7f64bae95640 1 -- 172.21.15.47:0/1773340849 wait complete. 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 10 monclient: build_initial_monmap 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 10 monclient: monmap: 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:12.450 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 1 librados: starting msgr at 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 1 librados: starting objecter 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 1 Processor -- start 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 1 -- start start 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 1 librados: setting wanted keys 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 1 librados: calling monclient init 2024-03-25T19:36:12.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.450+0000 7f64bae95640 10 monclient: init 2024-03-25T19:36:12.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:12.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:12.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient(hunting): picked mon.noname-a con 0x7f64b40b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:12.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient(hunting): picked mon.noname-b con 0x7f64b40b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:12.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient(hunting): picked mon.noname-c con 0x7f64b414be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:12.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b4142110 con 0x7f64b40b1730 2024-03-25T19:36:12.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b41d7830 con 0x7f64b40b4d60 2024-03-25T19:36:12.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b41d7b10 con 0x7f64b414be90 2024-03-25T19:36:12.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:12.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.451+0000 7f64bae95640 10 monclient(hunting): authenticate will time out at 5687.737401s 2024-03-25T19:36:12.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.460+0000 7f64b3fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f64b40b4d60 0x7f64b41d8010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:60716/0 (socket says 172.21.15.47:60716) 2024-03-25T19:36:12.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.460+0000 7f64b3fff640 1 -- 172.21.15.47:0/603489106 learned_addr learned my addr 172.21.15.47:0/603489106 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:12.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.463+0000 7f64b17fa640 10 client.?.objecter ms_handle_connect 0x7f64b40b4d60 2024-03-25T19:36:12.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.463+0000 7f64b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.468+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541265806 0 0) 0x7f64b41d7830 con 0x7f64b40b4d60 2024-03-25T19:36:12.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.468+0000 7f64b17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.468+0000 7f64b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.468+0000 7f64b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.468+0000 7f64b17fa640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:36:12.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.468+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498004eb0 con 0x7f64b40b4d60 2024-03-25T19:36:12.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480195610 0 0) 0x7f6498004eb0 con 0x7f64b40b4d60 2024-03-25T19:36:12.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b41d7830 con 0x7f64b40b4d60 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f64a0001690 con 0x7f64b40b4d60 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 10 monclient(hunting): dump: 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:12.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.478+0000 7f64b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:12.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.480+0000 7f64b17fa640 10 client.?.objecter ms_handle_connect 0x7f64b414be90 2024-03-25T19:36:12.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.480+0000 7f64b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.486+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552511086 0 0) 0x7f64b41d7b10 con 0x7f64b414be90 2024-03-25T19:36:12.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.486+0000 7f64b17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.486+0000 7f64b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.486+0000 7f64b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.486+0000 7f64b17fa640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:36:12.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.486+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498004eb0 con 0x7f64b414be90 2024-03-25T19:36:12.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.489+0000 7f64b17fa640 10 client.?.objecter ms_handle_connect 0x7f64b40b1730 2024-03-25T19:36:12.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.489+0000 7f64b17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1289780479 0 0) 0x7f64b41d7830 con 0x7f64b40b4d60 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 >> v1:172.21.15.47:6790/0 conn(0x7f64b414be90 legacy=0x7f64b4136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 >> v1:172.21.15.47:6789/0 conn(0x7f64b40b1730 legacy=0x7f64b4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 1 monclient: found mon.b 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b41382a0 con 0x7f64b40b4d60 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:36:12.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 10 monclient: _renew_subs 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b4138510 con 0x7f64b40b4d60 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 10 monclient: _renew_subs 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b4137630 con 0x7f64b40b4d60 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.491+0000 7f64bae95640 1 librados: init done 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.492+0000 7f64bae95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.492+0000 7f64bae95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.492+0000 7f64bae95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:12.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.492+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b41d7ca0 con 0x7f64b40b4d60 2024-03-25T19:36:12.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a00038a0 con 0x7f64b40b4d60 2024-03-25T19:36:12.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:12.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f64a0003cf0 con 0x7f64b40b4d60 2024-03-25T19:36:12.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 10 monclient: dump: 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:12.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b2ffd640 4 set_mon_vals no callback set 2024-03-25T19:36:12.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.783+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f64a0003f10 con 0x7f64b40b4d60 2024-03-25T19:36:12.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.784+0000 7f64b8c31640 10 monclient: get_auth_request con 0x7f649806dd40 auth_method 0 2024-03-25T19:36:12.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.784+0000 7f64b17fa640 10 client.4291.objecter ms_handle_connect 0x7f649806dd40 2024-03-25T19:36:12.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f64a0054a50 con 0x7f64b40b4d60 2024-03-25T19:36:12.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 10 client.4291.objecter ms_dispatch 0x7f64b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:12.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:12.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:12.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T19:36:12.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64a0050560 con 0x7f64b40b4d60 2024-03-25T19:36:12.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:12.789+0000 7f64b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:13.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.049+0000 7f64bae95640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:13.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.049+0000 7f64bae95640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:13.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.049+0000 7f64bae95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:13.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.049+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64b40627b0 con 0x7f64b40b4d60 2024-03-25T19:36:13.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.080+0000 7f64b17fa640 1 -- 172.21.15.47:0/603489106 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f64a0052490 con 0x7f64b40b4d60 2024-03-25T19:36:13.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.080+0000 7f64b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:13.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.080+0000 7f64b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:13.080 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:13.080 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:13.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 10 librados: watch_flush enter 2024-03-25T19:36:13.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 10 librados: watch_flush exit 2024-03-25T19:36:13.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T19:36:13.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 10 client.4291.objecter successfully canceled tick 2024-03-25T19:36:13.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 >> v1:172.21.15.142:6800/1025302268 conn(0x7f649806dd40 legacy=0x7f6498070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 10 monclient: shutdown 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.083+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 >> v1:172.21.15.142:6789/0 conn(0x7f64b40b4d60 legacy=0x7f64b41d8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.084+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 shutdown_connections 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.084+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 >> 172.21.15.47:0/603489106 conn(0x7f64b40617a0 msgr2=0x7f64b40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.084+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 shutdown_connections 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.084+0000 7f64bae95640 1 -- 172.21.15.47:0/603489106 wait complete. 2024-03-25T19:36:13.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:13.084+0000 7f64bae95640 1 librados: shutdown 2024-03-25T19:36:13.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:17.311 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:54.547 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.321 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:54.557 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.359 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:54.595 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.380 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:54.570 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.384 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:54.574 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.386 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:54.622 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.398 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:54.634 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.408 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:01:54.598 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.422 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:54.658 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:19.901 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.015+0000 7fbfc73af640 10 monclient: get_monmap_and_config 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.015+0000 7fbfc73af640 10 monclient: build_initial_monmap 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.015+0000 7fbfc73af640 10 monclient: monmap: 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:20.016487+0000 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:20.016487+0000 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:20.015 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:20.016 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.016+0000 7fbfc73af640 1 Processor -- start 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.016+0000 7fbfc73af640 1 -- start start 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.016+0000 7fbfc73af640 10 monclient: init 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient(hunting): picked mon.noname-c con 0x7fbfc00a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient(hunting): picked mon.noname-a con 0x7fbfc013f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient(hunting): picked mon.noname-b con 0x7fbfc01430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0142520 con 0x7fbfc013f710 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00b28a0 con 0x7fbfc00a98f0 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00b2a30 con 0x7fbfc01430c0 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:20.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.017+0000 7fbfc73af640 10 monclient(hunting): authenticate will time out at 5695.303533s 2024-03-25T19:36:20.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.024+0000 7fbfc514b640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbfc00a98f0 0x7fbfc00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:40188/0 (socket says 172.21.15.47:40188) 2024-03-25T19:36:20.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.024+0000 7fbfc514b640 1 -- 172.21.15.47:0/295224820 learned_addr learned my addr 172.21.15.47:0/295224820 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:20.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.041+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411670053 0 0) 0x7fbfc00b28a0 con 0x7fbfc00a98f0 2024-03-25T19:36:20.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.041+0000 7fbfb7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.041+0000 7fbfb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.041+0000 7fbfb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.041+0000 7fbfb7fff640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:36:20.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.041+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac003520 con 0x7fbfc00a98f0 2024-03-25T19:36:20.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.043+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241410606 0 0) 0x7fbfc00b2a30 con 0x7fbfc01430c0 2024-03-25T19:36:20.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.043+0000 7fbfb7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.043+0000 7fbfb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.043+0000 7fbfb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.043+0000 7fbfb7fff640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:36:20.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.043+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc00b28a0 con 0x7fbfc01430c0 2024-03-25T19:36:20.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283822408 0 0) 0x7fbfac003520 con 0x7fbfc00a98f0 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 >> v1:172.21.15.47:6790/0 conn(0x7fbfc01430c0 legacy=0x7fbfc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 >> v1:172.21.15.47:6789/0 conn(0x7fbfc013f710 legacy=0x7fbfc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 1 monclient: found mon.noname-c 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc00b44c0 con 0x7fbfc00a98f0 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbfbc001690 con 0x7fbfc00a98f0 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfb7fff640 10 monclient: dump: 2024-03-25T19:36:20.053 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfc73af640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:36:20.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.053+0000 7fbfc73af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfbc0017f0 con 0x7fbfc00a98f0 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfb7fff640 1 -- 172.21.15.47:0/295224820 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbfbc001c00 con 0x7fbfc00a98f0 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfb7fff640 10 monclient: dump: 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:20.060 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfc73af640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:20.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfc73af640 4 set_mon_vals no callback set 2024-03-25T19:36:20.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfc73af640 10 monclient: shutdown 2024-03-25T19:36:20.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfc73af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:20.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.061+0000 7fbfc73af640 1 -- 172.21.15.47:0/295224820 >> v1:172.21.15.142:6789/0 conn(0x7fbfc00a98f0 legacy=0x7fbfc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.062+0000 7fbfc73af640 1 -- 172.21.15.47:0/295224820 shutdown_connections 2024-03-25T19:36:20.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.062+0000 7fbfc73af640 1 -- 172.21.15.47:0/295224820 >> 172.21.15.47:0/295224820 conn(0x7fbfc00b7ea0 msgr2=0x7fbfc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:20.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.062+0000 7fbfc73af640 1 -- 172.21.15.47:0/295224820 shutdown_connections 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.062+0000 7fbfc73af640 1 -- 172.21.15.47:0/295224820 wait complete. 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.063+0000 7fbfc73af640 10 monclient: build_initial_monmap 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.063+0000 7fbfc73af640 10 monclient: monmap: 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:20.063 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.063+0000 7fbfc73af640 1 librados: starting msgr at 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.063+0000 7fbfc73af640 1 librados: starting objecter 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.063+0000 7fbfc73af640 1 Processor -- start 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 1 -- start start 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 1 librados: setting wanted keys 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 1 librados: calling monclient init 2024-03-25T19:36:20.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 10 monclient: init 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.064+0000 7fbfc73af640 10 monclient(hunting): picked mon.noname-c con 0x7fbfc00a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 10 monclient(hunting): picked mon.noname-a con 0x7fbfc013f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 10 monclient(hunting): picked mon.noname-b con 0x7fbfc01430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0136ee0 con 0x7fbfc013f710 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0137150 con 0x7fbfc01430c0 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc0137430 con 0x7fbfc00a98f0 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:20.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.065+0000 7fbfc73af640 10 monclient(hunting): authenticate will time out at 5695.351469s 2024-03-25T19:36:20.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.073+0000 7fbfc494a640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbfc013f710 0x7fbfc0137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58808/0 (socket says 172.21.15.47:58808) 2024-03-25T19:36:20.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.073+0000 7fbfc494a640 1 -- 172.21.15.47:0/2650290508 learned_addr learned my addr 172.21.15.47:0/2650290508 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:20.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.076+0000 7fbfb5ffb640 10 client.?.objecter ms_handle_connect 0x7fbfc013f710 2024-03-25T19:36:20.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.076+0000 7fbfb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:20.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.084+0000 7fbfb5ffb640 10 client.?.objecter ms_handle_connect 0x7fbfc01430c0 2024-03-25T19:36:20.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.084+0000 7fbfb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:20.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.088+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821094847 0 0) 0x7fbfc0136ee0 con 0x7fbfc013f710 2024-03-25T19:36:20.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.088+0000 7fbfb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.088+0000 7fbfb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.088+0000 7fbfb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.088+0000 7fbfb5ffb640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:36:20.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.088+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa4004eb0 con 0x7fbfc013f710 2024-03-25T19:36:20.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.091+0000 7fbfb5ffb640 10 client.?.objecter ms_handle_connect 0x7fbfc00a98f0 2024-03-25T19:36:20.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.091+0000 7fbfb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067882726 0 0) 0x7fbfa4004eb0 con 0x7fbfc013f710 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfc0136ee0 con 0x7fbfc013f710 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbfb00019d0 con 0x7fbfc013f710 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.100 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275856415 0 0) 0x7fbfc0137430 con 0x7fbfc00a98f0 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:36:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.100+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa4004eb0 con 0x7fbfc00a98f0 2024-03-25T19:36:20.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.105+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552690598 0 0) 0x7fbfc0137150 con 0x7fbfc01430c0 2024-03-25T19:36:20.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.105+0000 7fbfb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.105+0000 7fbfb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.105+0000 7fbfb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.105+0000 7fbfb5ffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:36:20.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.105+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0137430 con 0x7fbfc01430c0 2024-03-25T19:36:20.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2282994993 0 0) 0x7fbfc0136ee0 con 0x7fbfc013f710 2024-03-25T19:36:20.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 >> v1:172.21.15.47:6790/0 conn(0x7fbfc00a98f0 legacy=0x7fbfc00b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 >> v1:172.21.15.142:6789/0 conn(0x7fbfc01430c0 legacy=0x7fbfc0135070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:20.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 1 monclient: found mon.a 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0135970 con 0x7fbfc013f710 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 10 monclient: _renew_subs 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc0136220 con 0x7fbfc013f710 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 10 monclient: _renew_subs 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc01e36c0 con 0x7fbfc013f710 2024-03-25T19:36:20.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.110+0000 7fbfc73af640 1 librados: init done 2024-03-25T19:36:20.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.111+0000 7fbfc73af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.111+0000 7fbfc73af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.111+0000 7fbfc73af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:20.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.111+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc0135780 con 0x7fbfc013f710 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfb0001c50 con 0x7fbfc013f710 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fbfb0003a60 con 0x7fbfc013f710 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb5ffb640 10 monclient: dump: 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.430 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:20.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.430+0000 7fbfb77fe640 4 set_mon_vals no callback set 2024-03-25T19:36:20.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.431+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fbfb0003c80 con 0x7fbfc013f710 2024-03-25T19:36:20.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.431+0000 7fbfc514b640 10 monclient: get_auth_request con 0x7fbfa406de10 auth_method 0 2024-03-25T19:36:20.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.432+0000 7fbfb5ffb640 10 client.4305.objecter ms_handle_connect 0x7fbfa406de10 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fbfb0054880 con 0x7fbfc013f710 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 10 client.4305.objecter ms_dispatch 0x7fbfc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfb0050390 con 0x7fbfc013f710 2024-03-25T19:36:20.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.437+0000 7fbfb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:20.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.664+0000 7fbfc73af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.664+0000 7fbfc73af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.664+0000 7fbfc73af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:20.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.664+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfc0135b00 con 0x7fbfc013f710 2024-03-25T19:36:20.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.708+0000 7fbfb5ffb640 1 -- 172.21.15.47:0/2650290508 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fbfb00522c0 con 0x7fbfc013f710 2024-03-25T19:36:20.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.708+0000 7fbfb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.708+0000 7fbfb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:20.708 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:20.708 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:20.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.710+0000 7fbfc73af640 10 librados: watch_flush enter 2024-03-25T19:36:20.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.710+0000 7fbfc73af640 10 librados: watch_flush exit 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.710+0000 7fbfc73af640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.710+0000 7fbfc73af640 10 client.4305.objecter successfully canceled tick 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 >> v1:172.21.15.142:6800/1025302268 conn(0x7fbfa406de10 legacy=0x7fbfa40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 10 monclient: shutdown 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 >> v1:172.21.15.47:6789/0 conn(0x7fbfc013f710 legacy=0x7fbfc0137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 shutdown_connections 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 >> 172.21.15.47:0/2650290508 conn(0x7fbfc00b7ea0 msgr2=0x7fbfc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 shutdown_connections 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 -- 172.21.15.47:0/2650290508 wait complete. 2024-03-25T19:36:20.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:20.711+0000 7fbfc73af640 1 librados: shutdown 2024-03-25T19:36:20.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:27.529 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:27.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.644+0000 7f458f754640 10 monclient: get_monmap_and_config 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.644+0000 7f458f754640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.644+0000 7f458f754640 10 monclient: monmap: 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:27.645318+0000 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:27.645318+0000 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:27.644 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.645+0000 7f458f754640 1 Processor -- start 2024-03-25T19:36:27.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.645+0000 7f458f754640 1 -- start start 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.645+0000 7f458f754640 10 monclient: init 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient(hunting): picked mon.noname-b con 0x7f45880b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient(hunting): picked mon.noname-c con 0x7f45880b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient(hunting): picked mon.noname-a con 0x7f458814be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45880af940 con 0x7f458814be90 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45880b0a70 con 0x7f45880b1730 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45880b0d50 con 0x7f45880b4d60 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.646+0000 7f458f754640 10 monclient(hunting): authenticate will time out at 5702.932422s 2024-03-25T19:36:27.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.668+0000 7f458d4f0640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f45880b4d60 0x7f458813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:50074/0 (socket says 172.21.15.47:50074) 2024-03-25T19:36:27.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.668+0000 7f458d4f0640 1 -- 172.21.15.47:0/1056487432 learned_addr learned my addr 172.21.15.47:0/1056487432 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:27.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.680+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683577030 0 0) 0x7f45880b0d50 con 0x7f45880b4d60 2024-03-25T19:36:27.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.680+0000 7f457ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.680+0000 7f457ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.680+0000 7f457ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.680+0000 7f457ffff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:36:27.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.680+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4574003520 con 0x7f45880b4d60 2024-03-25T19:36:27.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.684+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345395857 0 0) 0x7f45880af940 con 0x7f458814be90 2024-03-25T19:36:27.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.684+0000 7f457ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.684+0000 7f457ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.684+0000 7f457ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.684+0000 7f457ffff640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:36:27.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.684+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45880b0d50 con 0x7f458814be90 2024-03-25T19:36:27.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.689+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056316428 0 0) 0x7f45880b0a70 con 0x7f45880b1730 2024-03-25T19:36:27.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.689+0000 7f457ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.689+0000 7f457ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.689+0000 7f457ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.689+0000 7f457ffff640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:36:27.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.689+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45880af940 con 0x7f45880b1730 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723747816 0 0) 0x7f4574003520 con 0x7f45880b4d60 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 >> v1:172.21.15.142:6789/0 conn(0x7f45880b1730 legacy=0x7f45880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 >> v1:172.21.15.47:6789/0 conn(0x7f458814be90 legacy=0x7f458814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 1 monclient: found mon.noname-b 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45880b1030 con 0x7f45880b4d60 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4584001690 con 0x7f45880b4d60 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f458f754640 5 monclient: authenticate success, global_id 4292 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f458f754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f457ffff640 10 monclient: dump: 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:27.691 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:27.692 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.690+0000 7f458f754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45840017f0 con 0x7f45880b4d60 2024-03-25T19:36:27.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f457ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f457ffff640 1 -- 172.21.15.47:0/1056487432 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4584001c00 con 0x7f45880b4d60 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f458f754640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f458f754640 4 set_mon_vals no callback set 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f457ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f457ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.695+0000 7f457ffff640 10 monclient: dump: 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:27.695 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:27.696 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:27.696 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 10 monclient: shutdown 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 1 -- 172.21.15.47:0/1056487432 >> v1:172.21.15.47:6790/0 conn(0x7f45880b4d60 legacy=0x7f458813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 1 -- 172.21.15.47:0/1056487432 shutdown_connections 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 1 -- 172.21.15.47:0/1056487432 >> 172.21.15.47:0/1056487432 conn(0x7f45880617a0 msgr2=0x7f4588061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 1 -- 172.21.15.47:0/1056487432 shutdown_connections 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.696+0000 7f458f754640 1 -- 172.21.15.47:0/1056487432 wait complete. 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 10 monclient: monmap: 2024-03-25T19:36:27.698 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 1 librados: starting msgr at 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 1 librados: starting objecter 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 1 Processor -- start 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 1 -- start start 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 1 librados: setting wanted keys 2024-03-25T19:36:27.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 1 librados: calling monclient init 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.697+0000 7f458f754640 10 monclient: init 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient(hunting): picked mon.noname-b con 0x7f45880b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient(hunting): picked mon.noname-a con 0x7f45880b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient(hunting): picked mon.noname-c con 0x7f458814be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4588137f30 con 0x7f45880b4d60 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4588131720 con 0x7f45880b1730 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45881dba00 con 0x7f458814be90 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.698+0000 7f458f754640 10 monclient(hunting): authenticate will time out at 5702.984648s 2024-03-25T19:36:27.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.703+0000 7f458ccef640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f45880b4d60 0x7f4588131010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53306/0 (socket says 172.21.15.47:53306) 2024-03-25T19:36:27.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.703+0000 7f458ccef640 1 -- 172.21.15.47:0/4021997964 learned_addr learned my addr 172.21.15.47:0/4021997964 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:27.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.706+0000 7f457dffb640 10 client.?.objecter ms_handle_connect 0x7f45880b4d60 2024-03-25T19:36:27.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.706+0000 7f457dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.713+0000 7f457dffb640 10 client.?.objecter ms_handle_connect 0x7f45880b1730 2024-03-25T19:36:27.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.713+0000 7f457dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.719+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706225556 0 0) 0x7f4588131720 con 0x7f45880b1730 2024-03-25T19:36:27.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.719+0000 7f457dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.719+0000 7f457dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.719+0000 7f457dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.719+0000 7f457dffb640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:36:27.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.719+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456c004eb0 con 0x7f45880b1730 2024-03-25T19:36:27.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.722+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401767190 0 0) 0x7f4588137f30 con 0x7f45880b4d60 2024-03-25T19:36:27.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.722+0000 7f457dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.722+0000 7f457dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.722+0000 7f457dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.722+0000 7f457dffb640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:36:27.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.722+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588131720 con 0x7f45880b4d60 2024-03-25T19:36:27.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.725+0000 7f457dffb640 10 client.?.objecter ms_handle_connect 0x7f458814be90 2024-03-25T19:36:27.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.725+0000 7f457dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338844695 0 0) 0x7f45881dba00 con 0x7f458814be90 2024-03-25T19:36:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588137f30 con 0x7f458814be90 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 121193549 0 0) 0x7f456c004eb0 con 0x7f45880b1730 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45881dba00 con 0x7f45880b1730 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4584001b00 con 0x7f45880b1730 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): dump: 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.735 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.734+0000 7f457dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:27.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.736+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3296495010 0 0) 0x7f4588131720 con 0x7f45880b4d60 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.736+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f456c004eb0 con 0x7f45880b4d60 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.736+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f45780019f0 con 0x7f45880b4d60 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.736+0000 7f457dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.736+0000 7f457dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.736+0000 7f457dffb640 10 monclient(hunting): dump: 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:27.737 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1005245212 0 0) 0x7f456c004eb0 con 0x7f45880b4d60 2024-03-25T19:36:27.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 >> v1:172.21.15.47:6790/0 conn(0x7f458814be90 legacy=0x7f45881db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 >> v1:172.21.15.142:6789/0 conn(0x7f45880b1730 legacy=0x7f4588137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 1 monclient: found mon.a 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45881dbb90 con 0x7f45880b4d60 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f457dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f458f754640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f458f754640 10 monclient: _renew_subs 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f458f754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.743+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4588143b60 con 0x7f45880b4d60 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.744+0000 7f458f754640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.744+0000 7f458f754640 10 monclient: _renew_subs 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.744+0000 7f458f754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.744+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4588136dd0 con 0x7f45880b4d60 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.744+0000 7f458f754640 1 librados: init done 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.745+0000 7f458f754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.745+0000 7f458f754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.745+0000 7f458f754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:27.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.745+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45881370d0 con 0x7f45880b4d60 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.767+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45780035f0 con 0x7f45880b4d60 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.768+0000 7f457dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.768+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f45780037a0 con 0x7f45880b4d60 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.768+0000 7f457dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.768+0000 7f457dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.768+0000 7f457dffb640 10 monclient: dump: 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:27.767 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:27.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.768+0000 7f457f7fe640 4 set_mon_vals no callback set 2024-03-25T19:36:27.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.769+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f45780039c0 con 0x7f45880b4d60 2024-03-25T19:36:27.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.769+0000 7f458d4f0640 10 monclient: get_auth_request con 0x7f456c06df60 auth_method 0 2024-03-25T19:36:27.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.770+0000 7f457dffb640 10 client.4311.objecter ms_handle_connect 0x7f456c06df60 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.772+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f4578054650 con 0x7f45880b4d60 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.772+0000 7f457dffb640 10 client.4311.objecter ms_dispatch 0x7f4588000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:27.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.772+0000 7f457dffb640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:27.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.772+0000 7f457dffb640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:27.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:27.772+0000 7f457dffb640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T19:36:28.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.076+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4578050160 con 0x7f45880b4d60 2024-03-25T19:36:28.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.076+0000 7f457dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:28.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.076+0000 7f457dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:28.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.329+0000 7f458f754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.329+0000 7f458f754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.329+0000 7f458f754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:28.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.329+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4588137370 con 0x7f45880b4d60 2024-03-25T19:36:28.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.385+0000 7f457dffb640 1 -- 172.21.15.47:0/4021997964 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f4588137370 con 0x7f45880b4d60 2024-03-25T19:36:28.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.385+0000 7f457dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.385+0000 7f457dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:28.385 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:28.385 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:28.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 10 librados: watch_flush enter 2024-03-25T19:36:28.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 10 librados: watch_flush exit 2024-03-25T19:36:28.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T19:36:28.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 10 client.4311.objecter successfully canceled tick 2024-03-25T19:36:28.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 >> v1:172.21.15.142:6800/1025302268 conn(0x7f456c06df60 legacy=0x7f456c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 10 monclient: shutdown 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 >> v1:172.21.15.47:6789/0 conn(0x7f45880b4d60 legacy=0x7f4588131010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 shutdown_connections 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 >> 172.21.15.47:0/4021997964 conn(0x7f45880617a0 msgr2=0x7f4588061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 shutdown_connections 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.388+0000 7f458f754640 1 -- 172.21.15.47:0/4021997964 wait complete. 2024-03-25T19:36:28.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:28.389+0000 7f458f754640 1 librados: shutdown 2024-03-25T19:36:28.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:35.206 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:35.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.322+0000 7f85fc997640 10 monclient: get_monmap_and_config 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.322+0000 7f85fc997640 10 monclient: build_initial_monmap 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.322+0000 7f85fc997640 10 monclient: monmap: 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:35.323344+0000 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:35.323344+0000 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:35.322 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:35.323 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.323+0000 7f85fc997640 1 Processor -- start 2024-03-25T19:36:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.323+0000 7f85fc997640 1 -- start start 2024-03-25T19:36:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.323+0000 7f85fc997640 10 monclient: init 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient(hunting): picked mon.noname-b con 0x7f85f80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient(hunting): picked mon.noname-c con 0x7f85f813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient(hunting): picked mon.noname-a con 0x7f85f81430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8142520 con 0x7f85f81430c0 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80b28a0 con 0x7f85f813f710 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80b2a30 con 0x7f85f80a98f0 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:35.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.324+0000 7f85fc997640 10 monclient(hunting): authenticate will time out at 5710.610850s 2024-03-25T19:36:35.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.342+0000 7f85f5d9b640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f85f813f710 0x7f85f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:59340/0 (socket says 172.21.15.47:59340) 2024-03-25T19:36:35.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.342+0000 7f85f5d9b640 1 -- 172.21.15.47:0/2036965218 learned_addr learned my addr 172.21.15.47:0/2036965218 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.353+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868048530 0 0) 0x7f85f80b28a0 con 0x7f85f813f710 2024-03-25T19:36:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.353+0000 7f85f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.353+0000 7f85f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.353+0000 7f85f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.353+0000 7f85f559a640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:36:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.353+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e0003520 con 0x7f85f813f710 2024-03-25T19:36:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.358+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31093452 0 0) 0x7f85f80b2a30 con 0x7f85f80a98f0 2024-03-25T19:36:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.358+0000 7f85f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.358+0000 7f85f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.358+0000 7f85f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.358+0000 7f85f559a640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:36:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.358+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f80b28a0 con 0x7f85f80a98f0 2024-03-25T19:36:35.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.363+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211336763 0 0) 0x7f85f8142520 con 0x7f85f81430c0 2024-03-25T19:36:35.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.363+0000 7f85f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.363+0000 7f85f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.363+0000 7f85f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.363+0000 7f85f559a640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:36:35.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.363+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f80b2a30 con 0x7f85f81430c0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1867058566 0 0) 0x7f85f80b28a0 con 0x7f85f80a98f0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 >> v1:172.21.15.142:6789/0 conn(0x7f85f813f710 legacy=0x7f85f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 >> v1:172.21.15.47:6789/0 conn(0x7f85f81430c0 legacy=0x7f85f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: _finish_hunting 0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 1 monclient: found mon.noname-b 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80b44c0 con 0x7f85f80a98f0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: _finish_auth 0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: _check_auth_tickets 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f85e8001670 con 0x7f85f80a98f0 2024-03-25T19:36:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85f559a640 10 monclient: dump: 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85fc997640 5 monclient: authenticate success, global_id 4298 2024-03-25T19:36:35.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.367+0000 7f85fc997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:35.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e80017d0 con 0x7f85f80a98f0 2024-03-25T19:36:35.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85f559a640 1 -- 172.21.15.47:0/2036965218 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f85e8001be0 con 0x7f85f80a98f0 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85f559a640 10 monclient: dump: 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:35.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85fc997640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:35.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.389+0000 7f85fc997640 4 set_mon_vals no callback set 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 10 monclient: shutdown 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 1 -- 172.21.15.47:0/2036965218 >> v1:172.21.15.47:6790/0 conn(0x7f85f80a98f0 legacy=0x7f85f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 1 -- 172.21.15.47:0/2036965218 shutdown_connections 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 1 -- 172.21.15.47:0/2036965218 >> 172.21.15.47:0/2036965218 conn(0x7f85f80b7ea0 msgr2=0x7f85f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 1 -- 172.21.15.47:0/2036965218 shutdown_connections 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.390+0000 7f85fc997640 1 -- 172.21.15.47:0/2036965218 wait complete. 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 10 monclient: build_initial_monmap 2024-03-25T19:36:35.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 10 monclient: monmap: 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 1 librados: starting msgr at 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 1 librados: starting objecter 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 1 Processor -- start 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 1 -- start start 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 1 librados: setting wanted keys 2024-03-25T19:36:35.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 1 librados: calling monclient init 2024-03-25T19:36:35.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.391+0000 7f85fc997640 10 monclient: init 2024-03-25T19:36:35.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient(hunting): picked mon.noname-b con 0x7f85f80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient(hunting): picked mon.noname-c con 0x7f85f813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient(hunting): picked mon.noname-a con 0x7f85f81430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80b7210 con 0x7f85f81430c0 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8138e60 con 0x7f85f80a98f0 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80b76a0 con 0x7f85f813f710 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:35.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.392+0000 7f85fc997640 10 monclient(hunting): authenticate will time out at 5710.678902s 2024-03-25T19:36:35.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.410+0000 7f85f659c640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f85f80a98f0 0x7f85f8139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:59392/0 (socket says 172.21.15.47:59392) 2024-03-25T19:36:35.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.411+0000 7f85f659c640 1 -- 172.21.15.47:0/2724064939 learned_addr learned my addr 172.21.15.47:0/2724064939 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:35.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.412+0000 7f85e6ffd640 10 client.?.objecter ms_handle_connect 0x7f85f80a98f0 2024-03-25T19:36:35.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.412+0000 7f85e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.416+0000 7f85e6ffd640 10 client.?.objecter ms_handle_connect 0x7f85f813f710 2024-03-25T19:36:35.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.416+0000 7f85e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.420+0000 7f85e6ffd640 10 client.?.objecter ms_handle_connect 0x7f85f81430c0 2024-03-25T19:36:35.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.420+0000 7f85e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.430+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589438481 0 0) 0x7f85f8138e60 con 0x7f85f80a98f0 2024-03-25T19:36:35.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.430+0000 7f85e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.430+0000 7f85e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.430+0000 7f85e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.430+0000 7f85e6ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:36:35.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.430+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d8004eb0 con 0x7f85f80a98f0 2024-03-25T19:36:35.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.434+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157421221 0 0) 0x7f85f80b76a0 con 0x7f85f813f710 2024-03-25T19:36:35.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.434+0000 7f85e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.434+0000 7f85e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.434+0000 7f85e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.434+0000 7f85e6ffd640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:36:35.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.434+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8138e60 con 0x7f85f813f710 2024-03-25T19:36:35.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.436+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134358776 0 0) 0x7f85f80b7210 con 0x7f85f81430c0 2024-03-25T19:36:35.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.436+0000 7f85e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.436+0000 7f85e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.436+0000 7f85e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.436+0000 7f85e6ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:36:35.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.436+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f80b76a0 con 0x7f85f81430c0 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 25059640 0 0) 0x7f85f80b76a0 con 0x7f85f81430c0 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85f80b7210 con 0x7f85f81430c0 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f85f0001bd0 con 0x7f85f81430c0 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.468+0000 7f85e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:35.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 323153382 0 0) 0x7f85f80b7210 con 0x7f85f81430c0 2024-03-25T19:36:35.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 >> v1:172.21.15.47:6790/0 conn(0x7f85f813f710 legacy=0x7f85f81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 >> v1:172.21.15.142:6789/0 conn(0x7f85f80a98f0 legacy=0x7f85f8139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 1 monclient: found mon.a 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f813b4d0 con 0x7f85f81430c0 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85fc997640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85fc997640 10 monclient: _renew_subs 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85fc997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.539+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f813b7b0 con 0x7f85f81430c0 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.540+0000 7f85fc997640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:35.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.540+0000 7f85fc997640 10 monclient: _renew_subs 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.540+0000 7f85fc997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.540+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f8130190 con 0x7f85f81430c0 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.540+0000 7f85fc997640 1 librados: init done 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.541+0000 7f85fc997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.541+0000 7f85fc997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.541+0000 7f85fc997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:35.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.541+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f80627b0 con 0x7f85f81430c0 2024-03-25T19:36:35.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85f0001fe0 con 0x7f85f81430c0 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f85f0003b30 con 0x7f85f81430c0 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85e6ffd640 10 monclient: dump: 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:35.901 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:35.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.901+0000 7f85f4d99640 4 set_mon_vals no callback set 2024-03-25T19:36:35.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.902+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f85f0019b00 con 0x7f85f81430c0 2024-03-25T19:36:35.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.902+0000 7f85f659c640 10 monclient: get_auth_request con 0x7f85d806ddd0 auth_method 0 2024-03-25T19:36:35.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.903+0000 7f85e6ffd640 10 client.4317.objecter ms_handle_connect 0x7f85d806ddd0 2024-03-25T19:36:35.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f85f00537d0 con 0x7f85f81430c0 2024-03-25T19:36:35.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 10 client.4317.objecter ms_dispatch 0x7f85f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:35.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:35.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:35.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:36:35.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85f004f110 con 0x7f85f81430c0 2024-03-25T19:36:35.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:35.908+0000 7f85e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:36.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.156+0000 7f85fc997640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:36.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.156+0000 7f85fc997640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:36.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.156+0000 7f85fc997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:36.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.156+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f813d640 con 0x7f85f81430c0 2024-03-25T19:36:36.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.199+0000 7f85e6ffd640 1 -- 172.21.15.47:0/2724064939 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f85f0051210 con 0x7f85f81430c0 2024-03-25T19:36:36.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.199+0000 7f85e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:36.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.199+0000 7f85e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:36.198 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:36.198 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:36.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 10 librados: watch_flush enter 2024-03-25T19:36:36.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 10 librados: watch_flush exit 2024-03-25T19:36:36.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:36:36.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 >> v1:172.21.15.142:6800/1025302268 conn(0x7f85d806ddd0 legacy=0x7f85d8070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 10 monclient: shutdown 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.201+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 >> v1:172.21.15.47:6789/0 conn(0x7f85f81430c0 legacy=0x7f85f81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.202+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 shutdown_connections 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.202+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 >> 172.21.15.47:0/2724064939 conn(0x7f85f80b7ea0 msgr2=0x7f85f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.202+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 shutdown_connections 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.202+0000 7f85fc997640 1 -- 172.21.15.47:0/2724064939 wait complete. 2024-03-25T19:36:36.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:36.202+0000 7f85fc997640 1 librados: shutdown 2024-03-25T19:36:36.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:37.361 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:14.596 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.364 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:14.599 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.396 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:14.632 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.414 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:14.605 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.421 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:14.611 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.424 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:14.659 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.431 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:14.667 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.468 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:14.704 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.474 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:14.664 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.019 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.134+0000 7f9dd3a58640 10 monclient: get_monmap_and_config 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.134+0000 7f9dd3a58640 10 monclient: build_initial_monmap 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.134+0000 7f9dd3a58640 10 monclient: monmap: 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:43.136371+0000 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:43.136371+0000 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:43.135 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:43.136 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:43.136 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:43.136 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.135+0000 7f9dd3a58640 1 Processor -- start 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.135+0000 7f9dd3a58640 1 -- start start 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.135+0000 7f9dd3a58640 10 monclient: init 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient(hunting): picked mon.noname-b con 0x7f9dcc0b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient(hunting): picked mon.noname-c con 0x7f9dcc0b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient(hunting): picked mon.noname-a con 0x7f9dcc14be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0af940 con 0x7f9dcc14be90 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0b0a70 con 0x7f9dcc0b1730 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0b0d50 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:43.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.136+0000 7f9dd3a58640 10 monclient(hunting): authenticate will time out at 5718.423234s 2024-03-25T19:36:43.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.152+0000 7f9dd17f4640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f9dcc0b4d60 0x7f9dcc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44744/0 (socket says 172.21.15.47:44744) 2024-03-25T19:36:43.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.152+0000 7f9dd17f4640 1 -- 172.21.15.47:0/2303829830 learned_addr learned my addr 172.21.15.47:0/2303829830 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:43.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.162+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475803446 0 0) 0x7f9dcc0b0a70 con 0x7f9dcc0b1730 2024-03-25T19:36:43.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.162+0000 7f9dc3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.162+0000 7f9dc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.162+0000 7f9dc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.162+0000 7f9dc3fff640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:36:43.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.162+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db8003520 con 0x7f9dcc0b1730 2024-03-25T19:36:43.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.166+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904735375 0 0) 0x7f9dcc0b0d50 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.166+0000 7f9dc3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.166+0000 7f9dc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.166+0000 7f9dc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.166+0000 7f9dc3fff640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:36:43.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.166+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc0b0a70 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.173+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469876092 0 0) 0x7f9dcc0af940 con 0x7f9dcc14be90 2024-03-25T19:36:43.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.173+0000 7f9dc3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.173+0000 7f9dc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.173+0000 7f9dc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.173+0000 7f9dc3fff640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:36:43.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.173+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc0b0d50 con 0x7f9dcc14be90 2024-03-25T19:36:43.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522377646 0 0) 0x7f9dcc0b0a70 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 >> v1:172.21.15.142:6789/0 conn(0x7f9dcc0b1730 legacy=0x7f9dcc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 >> v1:172.21.15.47:6789/0 conn(0x7f9dcc14be90 legacy=0x7f9dcc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 1 monclient: found mon.noname-b 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc0b1030 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9dc8001670 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dc3fff640 10 monclient: dump: 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:43.176 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dd3a58640 5 monclient: authenticate success, global_id 4304 2024-03-25T19:36:43.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.175+0000 7f9dd3a58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:43.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dc80017d0 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:43.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dc3fff640 1 -- 172.21.15.47:0/2303829830 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9dc8001be0 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:43.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 4 set_mon_vals no callback set 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dc3fff640 10 monclient: dump: 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:43.192 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 10 monclient: shutdown 2024-03-25T19:36:43.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:43.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 1 -- 172.21.15.47:0/2303829830 >> v1:172.21.15.47:6790/0 conn(0x7f9dcc0b4d60 legacy=0x7f9dcc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 1 -- 172.21.15.47:0/2303829830 shutdown_connections 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.191+0000 7f9dd3a58640 1 -- 172.21.15.47:0/2303829830 >> 172.21.15.47:0/2303829830 conn(0x7f9dcc0617a0 msgr2=0x7f9dcc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.192+0000 7f9dd3a58640 1 -- 172.21.15.47:0/2303829830 shutdown_connections 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.192+0000 7f9dd3a58640 1 -- 172.21.15.47:0/2303829830 wait complete. 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.192+0000 7f9dd3a58640 10 monclient: build_initial_monmap 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.192+0000 7f9dd3a58640 10 monclient: monmap: 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:43.195 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 librados: starting msgr at 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 librados: starting objecter 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 Processor -- start 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 -- start start 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 librados: setting wanted keys 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 librados: calling monclient init 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient: init 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient(hunting): picked mon.noname-a con 0x7f9dcc0b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient(hunting): picked mon.noname-b con 0x7f9dcc0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient(hunting): picked mon.noname-c con 0x7f9dcc14be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:43.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc145370 con 0x7f9dcc0b1730 2024-03-25T19:36:43.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc1455c0 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc1458a0 con 0x7f9dcc14be90 2024-03-25T19:36:43.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:43.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.193+0000 7f9dd3a58640 10 monclient(hunting): authenticate will time out at 5718.480760s 2024-03-25T19:36:43.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.208+0000 7f9dd1ff5640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f9dcc14be90 0x7f9dcc13d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44802/0 (socket says 172.21.15.47:44802) 2024-03-25T19:36:43.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.208+0000 7f9dd1ff5640 1 -- 172.21.15.47:0/3533896502 learned_addr learned my addr 172.21.15.47:0/3533896502 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:43.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.209+0000 7f9dc1ffb640 10 client.?.objecter ms_handle_connect 0x7f9dcc14be90 2024-03-25T19:36:43.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.209+0000 7f9dc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:43.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.211+0000 7f9dc1ffb640 10 client.?.objecter ms_handle_connect 0x7f9dcc0b1730 2024-03-25T19:36:43.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.211+0000 7f9dc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:43.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.218+0000 7f9dc1ffb640 10 client.?.objecter ms_handle_connect 0x7f9dcc0b4d60 2024-03-25T19:36:43.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.218+0000 7f9dc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:43.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.219+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272387956 0 0) 0x7f9dcc145370 con 0x7f9dcc0b1730 2024-03-25T19:36:43.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.219+0000 7f9dc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.219+0000 7f9dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.219+0000 7f9dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.219+0000 7f9dc1ffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:36:43.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.219+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db0004eb0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.229+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619960816 0 0) 0x7f9dcc1455c0 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.229+0000 7f9dc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.229+0000 7f9dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.229+0000 7f9dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.229+0000 7f9dc1ffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:36:43.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.229+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc145370 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.231+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834491565 0 0) 0x7f9dcc1458a0 con 0x7f9dcc14be90 2024-03-25T19:36:43.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.231+0000 7f9dc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.231+0000 7f9dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.231+0000 7f9dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.231+0000 7f9dc1ffb640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:36:43.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.231+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc1455c0 con 0x7f9dcc14be90 2024-03-25T19:36:43.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705401005 0 0) 0x7f9db0004eb0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dcc1458a0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9dc8001ba0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:43.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.234+0000 7f9dc1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.255+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2584304440 0 0) 0x7f9dcc145370 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.255+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9db0004eb0 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.255+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9dbc001950 con 0x7f9dcc0b4d60 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.255+0000 7f9dc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.255+0000 7f9dc1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.255+0000 7f9dc1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:43.255 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:43.256 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2976463001 0 0) 0x7f9dcc1458a0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 >> v1:172.21.15.47:6790/0 conn(0x7f9dcc14be90 legacy=0x7f9dcc13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 >> v1:172.21.15.142:6789/0 conn(0x7f9dcc0b4d60 legacy=0x7f9dcc13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 1 monclient: found mon.a 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:43.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc13f030 con 0x7f9dcc0b1730 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 10 monclient: _renew_subs 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc143900 con 0x7f9dcc0b1730 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 10 monclient: _renew_subs 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc13e0f0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.261+0000 7f9dd3a58640 1 librados: init done 2024-03-25T19:36:43.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.262+0000 7f9dd3a58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.262+0000 7f9dd3a58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.262+0000 7f9dd3a58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:43.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.262+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc13e340 con 0x7f9dcc0b1730 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dc8004a90 con 0x7f9dcc0b1730 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9dc8004da0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc1ffb640 10 monclient: dump: 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:43.575 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.574+0000 7f9dc37fe640 4 set_mon_vals no callback set 2024-03-25T19:36:43.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.575+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f9dc801ad70 con 0x7f9dcc0b1730 2024-03-25T19:36:43.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.575+0000 7f9dd0ff3640 10 monclient: get_auth_request con 0x7f9db006df70 auth_method 0 2024-03-25T19:36:43.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.576+0000 7f9dc1ffb640 10 client.4323.objecter ms_handle_connect 0x7f9db006df70 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f9dc8054910 con 0x7f9dcc0b1730 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 10 client.4323.objecter ms_dispatch 0x7f9dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9dc8050420 con 0x7f9dcc0b1730 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.581+0000 7f9dc1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:43.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.799+0000 7f9dd3a58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.799+0000 7f9dd3a58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.799+0000 7f9dd3a58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:36:43.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.799+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dcc0627b0 con 0x7f9dcc0b1730 2024-03-25T19:36:43.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.852+0000 7f9dc1ffb640 1 -- 172.21.15.47:0/3533896502 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f9dc8052350 con 0x7f9dcc0b1730 2024-03-25T19:36:43.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.853+0000 7f9dc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.853+0000 7f9dc1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:43.853 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:43.853 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 10 librados: watch_flush enter 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 10 librados: watch_flush exit 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 10 client.4323.objecter successfully canceled tick 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 >> v1:172.21.15.142:6800/1025302268 conn(0x7f9db006df70 legacy=0x7f9db0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 10 monclient: shutdown 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 >> v1:172.21.15.47:6789/0 conn(0x7f9dcc0b1730 legacy=0x7f9dcc140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 shutdown_connections 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 >> 172.21.15.47:0/3533896502 conn(0x7f9dcc0617a0 msgr2=0x7f9dcc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 shutdown_connections 2024-03-25T19:36:43.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 -- 172.21.15.47:0/3533896502 wait complete. 2024-03-25T19:36:43.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:43.855+0000 7f9dd3a58640 1 librados: shutdown 2024-03-25T19:36:43.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:50.674 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.780+0000 7fa6dfd6d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.780+0000 7fa6dfd6d640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.780+0000 7fa6dfd6d640 10 monclient: monmap: 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:50.781417+0000 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:50.781417+0000 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:50.780 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:50.781 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:50.781 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.781+0000 7fa6dfd6d640 1 Processor -- start 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.781+0000 7fa6dfd6d640 1 -- start start 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.781+0000 7fa6dfd6d640 10 monclient: init 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.781+0000 7fa6dfd6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.781+0000 7fa6dfd6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:50.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:50.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8142520 con 0x7fa6d80a98f0 2024-03-25T19:36:50.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80b28a0 con 0x7fa6d81430c0 2024-03-25T19:36:50.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80b2a30 con 0x7fa6d813f710 2024-03-25T19:36:50.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.782+0000 7fa6dfd6d640 10 monclient(hunting): authenticate will time out at 5726.068037s 2024-03-25T19:36:50.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.786+0000 7fa6de30a640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa6d81430c0 0x7fa6d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:43592/0 (socket says 172.21.15.47:43592) 2024-03-25T19:36:50.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.786+0000 7fa6de30a640 1 -- 172.21.15.47:0/1176014281 learned_addr learned my addr 172.21.15.47:0/1176014281 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:50.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.807+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867324881 0 0) 0x7fa6d8142520 con 0x7fa6d80a98f0 2024-03-25T19:36:50.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.807+0000 7fa6dcb07640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.807+0000 7fa6dcb07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.807+0000 7fa6dcb07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.807+0000 7fa6dcb07640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:36:50.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.807+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4003520 con 0x7fa6d80a98f0 2024-03-25T19:36:50.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.809+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303961777 0 0) 0x7fa6d80b28a0 con 0x7fa6d81430c0 2024-03-25T19:36:50.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.809+0000 7fa6dcb07640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.809+0000 7fa6dcb07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.809+0000 7fa6dcb07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.809+0000 7fa6dcb07640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:36:50.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.809+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8142520 con 0x7fa6d81430c0 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034968781 0 0) 0x7fa6c4003520 con 0x7fa6d80a98f0 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 >> v1:172.21.15.47:6790/0 conn(0x7fa6d813f710 legacy=0x7fa6d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 >> v1:172.21.15.142:6789/0 conn(0x7fa6d81430c0 legacy=0x7fa6d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 1 monclient: found mon.noname-a 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80b44c0 con 0x7fa6d80a98f0 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa6d4001690 con 0x7fa6d80a98f0 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dcb07640 10 monclient: dump: 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:50.817 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:50.818 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:50.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dfd6d640 5 monclient: authenticate success, global_id 4326 2024-03-25T19:36:50.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.816+0000 7fa6dfd6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:50.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d40017f0 con 0x7fa6d80a98f0 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dcb07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dcb07640 1 -- 172.21.15.47:0/1176014281 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa6d4001c00 con 0x7fa6d80a98f0 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dfd6d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dfd6d640 4 set_mon_vals no callback set 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dcb07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dcb07640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.819+0000 7fa6dcb07640 10 monclient: dump: 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.819 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:50.820 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:50.820 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:50.820 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:50.820 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 10 monclient: shutdown 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/1176014281 >> v1:172.21.15.47:6789/0 conn(0x7fa6d80a98f0 legacy=0x7fa6d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/1176014281 shutdown_connections 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/1176014281 >> 172.21.15.47:0/1176014281 conn(0x7fa6d80b7ea0 msgr2=0x7fa6d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/1176014281 shutdown_connections 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.820+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/1176014281 wait complete. 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 10 monclient: monmap: 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:50.822 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 1 librados: starting msgr at 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 1 librados: starting objecter 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 1 Processor -- start 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 1 -- start start 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 1 librados: setting wanted keys 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 1 librados: calling monclient init 2024-03-25T19:36:50.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.821+0000 7fa6dfd6d640 10 monclient: init 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8132a30 con 0x7fa6d813f710 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8138c80 con 0x7fa6d80a98f0 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8138f60 con 0x7fa6d81430c0 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.822+0000 7fa6dfd6d640 10 monclient(hunting): authenticate will time out at 5726.108380s 2024-03-25T19:36:50.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.828+0000 7fa6ddb09640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa6d80a98f0 0x7fa6d8137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:43622/0 (socket says 172.21.15.47:43622) 2024-03-25T19:36:50.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.828+0000 7fa6ddb09640 1 -- 172.21.15.47:0/3031662989 learned_addr learned my addr 172.21.15.47:0/3031662989 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:50.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.836+0000 7fa6ce7fc640 10 client.?.objecter ms_handle_connect 0x7fa6d80a98f0 2024-03-25T19:36:50.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.836+0000 7fa6ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.847+0000 7fa6ce7fc640 10 client.?.objecter ms_handle_connect 0x7fa6d81430c0 2024-03-25T19:36:50.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.847+0000 7fa6ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.848+0000 7fa6ce7fc640 10 client.?.objecter ms_handle_connect 0x7fa6d813f710 2024-03-25T19:36:50.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.848+0000 7fa6ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.852+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58440932 0 0) 0x7fa6d8138c80 con 0x7fa6d80a98f0 2024-03-25T19:36:50.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.852+0000 7fa6ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.852+0000 7fa6ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.852+0000 7fa6ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.852+0000 7fa6ce7fc640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:36:50.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.852+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6bc004eb0 con 0x7fa6d80a98f0 2024-03-25T19:36:50.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.862+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990448930 0 0) 0x7fa6d8132a30 con 0x7fa6d813f710 2024-03-25T19:36:50.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.862+0000 7fa6ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.862+0000 7fa6ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.862+0000 7fa6ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.862+0000 7fa6ce7fc640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:36:50.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.862+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8138c80 con 0x7fa6d813f710 2024-03-25T19:36:50.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.868+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93821196 0 0) 0x7fa6d8138f60 con 0x7fa6d81430c0 2024-03-25T19:36:50.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.868+0000 7fa6ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.868+0000 7fa6ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.868+0000 7fa6ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.868+0000 7fa6ce7fc640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:36:50.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.868+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8132a30 con 0x7fa6d81430c0 2024-03-25T19:36:50.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237181577 0 0) 0x7fa6bc004eb0 con 0x7fa6d80a98f0 2024-03-25T19:36:50.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6d8138f60 con 0x7fa6d80a98f0 2024-03-25T19:36:50.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa6d4001690 con 0x7fa6d80a98f0 2024-03-25T19:36:50.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.872+0000 7fa6ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:50.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.888+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1232546339 0 0) 0x7fa6d8138c80 con 0x7fa6d813f710 2024-03-25T19:36:50.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.888+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6bc004eb0 con 0x7fa6d813f710 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.888+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa6c80018a0 con 0x7fa6d813f710 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.888+0000 7fa6ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.888+0000 7fa6ce7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.888+0000 7fa6ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:50.888 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:50.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2906431059 0 0) 0x7fa6d8138f60 con 0x7fa6d80a98f0 2024-03-25T19:36:50.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 >> v1:172.21.15.47:6790/0 conn(0x7fa6d81430c0 legacy=0x7fa6d8132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 >> v1:172.21.15.47:6789/0 conn(0x7fa6d813f710 legacy=0x7fa6d8131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 1 monclient: found mon.b 2024-03-25T19:36:50.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d8139240 con 0x7fa6d80a98f0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 10 monclient: _renew_subs 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d8136e50 con 0x7fa6d80a98f0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 10 monclient: _renew_subs 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d81389f0 con 0x7fa6d80a98f0 2024-03-25T19:36:50.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.894+0000 7fa6dfd6d640 1 librados: init done 2024-03-25T19:36:50.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.895+0000 7fa6dfd6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.895+0000 7fa6dfd6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.895+0000 7fa6dfd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:50.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:50.895+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d80627b0 con 0x7fa6d80a98f0 2024-03-25T19:36:51.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d40038a0 con 0x7fa6d80a98f0 2024-03-25T19:36:51.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:51.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa6d4001cc0 con 0x7fa6d80a98f0 2024-03-25T19:36:51.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:51.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6cffff640 4 set_mon_vals no callback set 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.172+0000 7fa6ce7fc640 10 monclient: dump: 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:51.173 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:51.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.173+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fa6d4003db0 con 0x7fa6d80a98f0 2024-03-25T19:36:51.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.174+0000 7fa6dd308640 10 monclient: get_auth_request con 0x7fa6bc06df70 auth_method 0 2024-03-25T19:36:51.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.174+0000 7fa6ce7fc640 10 client.4321.objecter ms_handle_connect 0x7fa6bc06df70 2024-03-25T19:36:51.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.178+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fa6d4054950 con 0x7fa6d80a98f0 2024-03-25T19:36:51.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.178+0000 7fa6ce7fc640 10 client.4321.objecter ms_dispatch 0x7fa6d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:51.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.178+0000 7fa6ce7fc640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:51.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.178+0000 7fa6ce7fc640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:51.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.178+0000 7fa6ce7fc640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:36:51.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.179+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6d4050460 con 0x7fa6d80a98f0 2024-03-25T19:36:51.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.179+0000 7fa6ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:51.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.179+0000 7fa6ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:51.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.430+0000 7fa6dfd6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.431+0000 7fa6dfd6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.431+0000 7fa6dfd6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:51.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.431+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6d813dd60 con 0x7fa6d80a98f0 2024-03-25T19:36:51.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.458+0000 7fa6ce7fc640 1 -- 172.21.15.47:0/3031662989 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fa6d4052390 con 0x7fa6d80a98f0 2024-03-25T19:36:51.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.458+0000 7fa6ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.458+0000 7fa6ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:51.457 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:51.457 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:51.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 10 librados: watch_flush enter 2024-03-25T19:36:51.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 10 librados: watch_flush exit 2024-03-25T19:36:51.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:36:51.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:36:51.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 >> v1:172.21.15.142:6800/1025302268 conn(0x7fa6bc06df70 legacy=0x7fa6bc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:51.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 10 monclient: shutdown 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 >> v1:172.21.15.142:6789/0 conn(0x7fa6d80a98f0 legacy=0x7fa6d8137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 shutdown_connections 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 >> 172.21.15.47:0/3031662989 conn(0x7fa6d80b7ea0 msgr2=0x7fa6d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 shutdown_connections 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 -- 172.21.15.47:0/3031662989 wait complete. 2024-03-25T19:36:51.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:51.460+0000 7fa6dfd6d640 1 librados: shutdown 2024-03-25T19:36:51.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:34.614 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:34.664 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.472 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:34.707 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.483 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:34.719 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.485 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:34.676 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.490 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:34.680 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.497 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:34.733 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.509 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:34.698 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:57.523 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:34.759 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.279 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:58.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.395+0000 7f9472e40640 10 monclient: get_monmap_and_config 2024-03-25T19:36:58.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.395+0000 7f9472e40640 10 monclient: build_initial_monmap 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.395+0000 7f9472e40640 10 monclient: monmap: 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:36:58.396091+0000 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:36:58.396091+0000 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:36:58.395 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.396+0000 7f9472e40640 1 Processor -- start 2024-03-25T19:36:58.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.396+0000 7f9472e40640 1 -- start start 2024-03-25T19:36:58.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.396+0000 7f9472e40640 10 monclient: init 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.396+0000 7f9472e40640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 10 monclient(hunting): picked mon.noname-a con 0x7f946c06ab80 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 10 monclient(hunting): picked mon.noname-c con 0x7f946c06af60 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 10 monclient(hunting): picked mon.noname-b con 0x7f946c13f990 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0afa60 con 0x7f946c06ab80 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0afce0 con 0x7f946c06af60 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0b1150 con 0x7f946c13f990 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:58.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.397+0000 7f9472e40640 10 monclient(hunting): authenticate will time out at 5733.683027s 2024-03-25T19:36:58.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.408+0000 7f9470bdc640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f946c06ab80 0x7f946c0724e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33300/0 (socket says 172.21.15.47:33300) 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.408+0000 7f9470bdc640 1 -- 172.21.15.47:0/3671166538 learned_addr learned my addr 172.21.15.47:0/3671166538 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:58.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.424+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785861746 0 0) 0x7f946c0afce0 con 0x7f946c06af60 2024-03-25T19:36:58.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.424+0000 7f946b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.424+0000 7f946b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.424+0000 7f946b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.424+0000 7f946b7fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:36:58.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.425+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c003520 con 0x7f946c06af60 2024-03-25T19:36:58.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.426+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439725253 0 0) 0x7f946c0afa60 con 0x7f946c06ab80 2024-03-25T19:36:58.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.426+0000 7f946b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.426+0000 7f946b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.426+0000 7f946b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.426+0000 7f946b7fe640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:36:58.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.426+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c0afce0 con 0x7f946c06ab80 2024-03-25T19:36:58.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.427+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161323734 0 0) 0x7f946c0b1150 con 0x7f946c13f990 2024-03-25T19:36:58.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.427+0000 7f946b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.427+0000 7f946b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.427+0000 7f946b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.427+0000 7f946b7fe640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:36:58.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.427+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c0afa60 con 0x7f946c13f990 2024-03-25T19:36:58.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022815403 0 0) 0x7f946c0afce0 con 0x7f946c06ab80 2024-03-25T19:36:58.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 >> v1:172.21.15.47:6790/0 conn(0x7f946c13f990 legacy=0x7f946c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 >> v1:172.21.15.142:6789/0 conn(0x7f946c06af60 legacy=0x7f946c1458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:58.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:36:58.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c0aff20 con 0x7f946c06ab80 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9460001670 con 0x7f946c06ab80 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f946b7fe640 10 monclient: dump: 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f9472e40640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.437+0000 7f9472e40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94600017d0 con 0x7f946c06ab80 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f946b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f946b7fe640 1 -- 172.21.15.47:0/3671166538 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9460001be0 con 0x7f946c06ab80 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f9472e40640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f9472e40640 4 set_mon_vals no callback set 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f946b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f946b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f946b7fe640 10 monclient: dump: 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:58.449 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f9472e40640 10 monclient: shutdown 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f9472e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.449+0000 7f9472e40640 1 -- 172.21.15.47:0/3671166538 >> v1:172.21.15.47:6789/0 conn(0x7f946c06ab80 legacy=0x7f946c0724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.450+0000 7f9472e40640 1 -- 172.21.15.47:0/3671166538 shutdown_connections 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.450+0000 7f9472e40640 1 -- 172.21.15.47:0/3671166538 >> 172.21.15.47:0/3671166538 conn(0x7f946c06c830 msgr2=0x7f946c0b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.450+0000 7f9472e40640 1 -- 172.21.15.47:0/3671166538 shutdown_connections 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.450+0000 7f9472e40640 1 -- 172.21.15.47:0/3671166538 wait complete. 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.450+0000 7f9472e40640 10 monclient: build_initial_monmap 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.450+0000 7f9472e40640 10 monclient: monmap: 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:36:58.451 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 1 librados: starting msgr at 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 1 librados: starting objecter 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 1 Processor -- start 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 1 -- start start 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 1 librados: setting wanted keys 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 1 librados: calling monclient init 2024-03-25T19:36:58.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.451+0000 7f9472e40640 10 monclient: init 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient(hunting): picked mon.noname-c con 0x7f946c06ab80 addr v1:172.21.15.47:6790/0 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient(hunting): picked mon.noname-b con 0x7f946c13f990 addr v1:172.21.15.142:6789/0 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient(hunting): picked mon.noname-a con 0x7f946c1408d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c1d8150 con 0x7f946c1408d0 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c1d8670 con 0x7f946c13f990 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c1d8800 con 0x7f946c06ab80 2024-03-25T19:36:58.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:58.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.452+0000 7f9472e40640 10 monclient(hunting): authenticate will time out at 5733.738477s 2024-03-25T19:36:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.467+0000 7f94713dd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f946c1408d0 0x7f946c1391a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33314/0 (socket says 172.21.15.47:33314) 2024-03-25T19:36:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.468+0000 7f94713dd640 1 -- 172.21.15.47:0/1572542397 learned_addr learned my addr 172.21.15.47:0/1572542397 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:36:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.468+0000 7f94697fa640 10 client.?.objecter ms_handle_connect 0x7f946c1408d0 2024-03-25T19:36:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.468+0000 7f94697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.469+0000 7f94697fa640 10 client.?.objecter ms_handle_connect 0x7f946c13f990 2024-03-25T19:36:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.469+0000 7f94697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.478+0000 7f94697fa640 10 client.?.objecter ms_handle_connect 0x7f946c06ab80 2024-03-25T19:36:58.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.478+0000 7f94697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.496+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196074052 0 0) 0x7f946c1d8670 con 0x7f946c13f990 2024-03-25T19:36:58.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.496+0000 7f94697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.496+0000 7f94697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.496+0000 7f94697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.496+0000 7f94697fa640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:36:58.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.496+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9450004eb0 con 0x7f946c13f990 2024-03-25T19:36:58.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.498+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630275855 0 0) 0x7f946c1d8150 con 0x7f946c1408d0 2024-03-25T19:36:58.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.498+0000 7f94697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.498+0000 7f94697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.498+0000 7f94697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.498+0000 7f94697fa640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:36:58.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.498+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c1d8670 con 0x7f946c1408d0 2024-03-25T19:36:58.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.502+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434625767 0 0) 0x7f946c1d8800 con 0x7f946c06ab80 2024-03-25T19:36:58.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.502+0000 7f94697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.502+0000 7f94697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.502+0000 7f94697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.502+0000 7f94697fa640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:36:58.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.502+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946c1d8150 con 0x7f946c06ab80 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.511+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625573609 0 0) 0x7f9450004eb0 con 0x7f946c13f990 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.512+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946c1d8800 con 0x7f946c13f990 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.512+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9458001950 con 0x7f946c13f990 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.512+0000 7f94697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.512+0000 7f94697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.512+0000 7f94697fa640 10 monclient(hunting): dump: 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:58.511 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.512+0000 7f94697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.522+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1564373085 0 0) 0x7f946c1d8670 con 0x7f946c1408d0 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.522+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9450004eb0 con 0x7f946c1408d0 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.522+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9464001c50 con 0x7f946c1408d0 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.522+0000 7f94697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.522+0000 7f94697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.522+0000 7f94697fa640 10 monclient(hunting): dump: 2024-03-25T19:36:58.521 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:58.522 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.525+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3199754227 0 0) 0x7f946c1d8150 con 0x7f946c06ab80 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.525+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946c1d8670 con 0x7f946c06ab80 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.525+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9460001ae0 con 0x7f946c06ab80 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.525+0000 7f94697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.525+0000 7f94697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.525+0000 7f94697fa640 10 monclient(hunting): dump: 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:58.525 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2138274407 0 0) 0x7f946c1d8800 con 0x7f946c13f990 2024-03-25T19:36:58.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 >> v1:172.21.15.47:6790/0 conn(0x7f946c06ab80 legacy=0x7f946c1d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 >> v1:172.21.15.47:6789/0 conn(0x7f946c1408d0 legacy=0x7f946c1391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:58.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 1 monclient: found mon.b 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c139e70 con 0x7f946c13f990 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f94697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 5 monclient: authenticate success, global_id 4327 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 10 monclient: _renew_subs 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c13a0c0 con 0x7f946c13f990 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 10 monclient: _renew_subs 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c139a90 con 0x7f946c13f990 2024-03-25T19:36:58.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.532+0000 7f9472e40640 1 librados: init done 2024-03-25T19:36:58.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.533+0000 7f9472e40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.533+0000 7f9472e40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.533+0000 7f9472e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:58.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.533+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c0627b0 con 0x7f946c13f990 2024-03-25T19:36:58.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.831+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9458003610 con 0x7f946c13f990 2024-03-25T19:36:58.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.831+0000 7f94697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.831+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9458001d80 con 0x7f946c13f990 2024-03-25T19:36:58.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.831+0000 7f94697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.832+0000 7f94697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:58.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.832+0000 7f94697fa640 10 monclient: dump: 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:36:58.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.831+0000 7f946affd640 4 set_mon_vals no callback set 2024-03-25T19:36:58.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.832+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f94580197d0 con 0x7f946c13f990 2024-03-25T19:36:58.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.833+0000 7f9470bdc640 10 monclient: get_auth_request con 0x7f945006e0b0 auth_method 0 2024-03-25T19:36:58.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.833+0000 7f94697fa640 10 client.4327.objecter ms_handle_connect 0x7f945006e0b0 2024-03-25T19:36:58.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f94580535a0 con 0x7f946c13f990 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 10 client.4327.objecter ms_dispatch 0x7f946c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f945804eee0 con 0x7f946c13f990 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:58.837+0000 7f94697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:59.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.053+0000 7f9472e40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:59.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.053+0000 7f9472e40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:59.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.053+0000 7f9472e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:36:59.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.053+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f946c13e7b0 con 0x7f946c13f990 2024-03-25T19:36:59.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.124+0000 7f94697fa640 1 -- 172.21.15.47:0/1572542397 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f9458050fe0 con 0x7f946c13f990 2024-03-25T19:36:59.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.124+0000 7f94697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:59.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.124+0000 7f94697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:59.124 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:36:59.124 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 10 librados: watch_flush enter 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 10 librados: watch_flush exit 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 10 client.4327.objecter successfully canceled tick 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 >> v1:172.21.15.142:6800/1025302268 conn(0x7f945006e0b0 legacy=0x7f9450070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 10 monclient: shutdown 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 >> v1:172.21.15.142:6789/0 conn(0x7f946c13f990 legacy=0x7f946c1401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 shutdown_connections 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 >> 172.21.15.47:0/1572542397 conn(0x7f946c06c830 msgr2=0x7f946c0b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 shutdown_connections 2024-03-25T19:36:59.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 -- 172.21.15.47:0/1572542397 wait complete. 2024-03-25T19:36:59.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:36:59.127+0000 7f9472e40640 1 librados: shutdown 2024-03-25T19:36:59.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:05.944 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:06.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.060+0000 7fb64742a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:06.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.060+0000 7fb64742a640 10 monclient: build_initial_monmap 2024-03-25T19:37:06.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.060+0000 7fb64742a640 10 monclient: monmap: 2024-03-25T19:37:06.060 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:06.061664+0000 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:06.061664+0000 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:06.061 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:06.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.061+0000 7fb64742a640 1 Processor -- start 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 1 -- start start 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient: init 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient(hunting): picked mon.noname-c con 0x7fb6400a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient(hunting): picked mon.noname-a con 0x7fb64013f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient(hunting): picked mon.noname-b con 0x7fb6401430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb640142520 con 0x7fb64013f710 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400b28a0 con 0x7fb6400a98f0 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400b2a30 con 0x7fb6401430c0 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.062+0000 7fb64742a640 10 monclient(hunting): authenticate will time out at 5741.348784s 2024-03-25T19:37:06.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.072+0000 7fb6451c6640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb6400a98f0 0x7fb6400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:48498/0 (socket says 172.21.15.47:48498) 2024-03-25T19:37:06.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.072+0000 7fb6451c6640 1 -- 172.21.15.47:0/1364991438 learned_addr learned my addr 172.21.15.47:0/1364991438 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:06.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.101+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264168657 0 0) 0x7fb6400b28a0 con 0x7fb6400a98f0 2024-03-25T19:37:06.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.101+0000 7fb637fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.101+0000 7fb637fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.101+0000 7fb637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.101+0000 7fb637fff640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:37:06.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.101+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c003520 con 0x7fb6400a98f0 2024-03-25T19:37:06.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.109+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675672615 0 0) 0x7fb6400b2a30 con 0x7fb6401430c0 2024-03-25T19:37:06.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.109+0000 7fb637fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.109+0000 7fb637fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.109+0000 7fb637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.109+0000 7fb637fff640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.109+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6400b28a0 con 0x7fb6401430c0 2024-03-25T19:37:06.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.112+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838225788 0 0) 0x7fb640142520 con 0x7fb64013f710 2024-03-25T19:37:06.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.112+0000 7fb637fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.112+0000 7fb637fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.112+0000 7fb637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.112+0000 7fb637fff640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:37:06.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.112+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6400b2a30 con 0x7fb64013f710 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840573461 0 0) 0x7fb62c003520 con 0x7fb6400a98f0 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 >> v1:172.21.15.47:6790/0 conn(0x7fb6401430c0 legacy=0x7fb64014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 >> v1:172.21.15.47:6789/0 conn(0x7fb64013f710 legacy=0x7fb6401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 1 monclient: found mon.noname-c 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400b44c0 con 0x7fb6400a98f0 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:06.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb63c001690 con 0x7fb6400a98f0 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb637fff640 10 monclient: dump: 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb64742a640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:37:06.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.119+0000 7fb64742a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:06.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb63c0017f0 con 0x7fb6400a98f0 2024-03-25T19:37:06.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb637fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb637fff640 1 -- 172.21.15.47:0/1364991438 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb63c001c00 con 0x7fb6400a98f0 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb637fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb637fff640 10 monclient: dump: 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:06.122 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:06.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb64742a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:06.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.122+0000 7fb64742a640 4 set_mon_vals no callback set 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 10 monclient: shutdown 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 1 -- 172.21.15.47:0/1364991438 >> v1:172.21.15.142:6789/0 conn(0x7fb6400a98f0 legacy=0x7fb6400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 1 -- 172.21.15.47:0/1364991438 shutdown_connections 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 1 -- 172.21.15.47:0/1364991438 >> 172.21.15.47:0/1364991438 conn(0x7fb6400b7ea0 msgr2=0x7fb6400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 1 -- 172.21.15.47:0/1364991438 shutdown_connections 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.123+0000 7fb64742a640 1 -- 172.21.15.47:0/1364991438 wait complete. 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.124+0000 7fb64742a640 10 monclient: build_initial_monmap 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.124+0000 7fb64742a640 10 monclient: monmap: 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.124+0000 7fb64742a640 1 librados: starting msgr at 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.124+0000 7fb64742a640 1 librados: starting objecter 2024-03-25T19:37:06.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.124+0000 7fb64742a640 1 Processor -- start 2024-03-25T19:37:06.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 1 -- start start 2024-03-25T19:37:06.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 1 librados: setting wanted keys 2024-03-25T19:37:06.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 1 librados: calling monclient init 2024-03-25T19:37:06.126 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient: init 2024-03-25T19:37:06.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:06.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:06.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient(hunting): picked mon.noname-c con 0x7fb6400a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:06.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient(hunting): picked mon.noname-b con 0x7fb64013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:06.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient(hunting): picked mon.noname-a con 0x7fb6401430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:06.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400704d0 con 0x7fb6401430c0 2024-03-25T19:37:06.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6401df0d0 con 0x7fb64013f710 2024-03-25T19:37:06.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64006f490 con 0x7fb6400a98f0 2024-03-25T19:37:06.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:06.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.125+0000 7fb64742a640 10 monclient(hunting): authenticate will time out at 5741.411725s 2024-03-25T19:37:06.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.137+0000 7fb6449c5640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb64013f710 0x7fb6401df5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:48574/0 (socket says 172.21.15.47:48574) 2024-03-25T19:37:06.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.137+0000 7fb6449c5640 1 -- 172.21.15.47:0/3848808412 learned_addr learned my addr 172.21.15.47:0/3848808412 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.139+0000 7fb635ffb640 10 client.?.objecter ms_handle_connect 0x7fb64013f710 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.139+0000 7fb635ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:06.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849489038 0 0) 0x7fb6401df0d0 con 0x7fb64013f710 2024-03-25T19:37:06.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:37:06.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb624004eb0 con 0x7fb64013f710 2024-03-25T19:37:06.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 10 client.?.objecter ms_handle_connect 0x7fb6400a98f0 2024-03-25T19:37:06.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.150+0000 7fb635ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:06.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.153+0000 7fb635ffb640 10 client.?.objecter ms_handle_connect 0x7fb6401430c0 2024-03-25T19:37:06.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.153+0000 7fb635ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1569194025 0 0) 0x7fb624004eb0 con 0x7fb64013f710 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6401df0d0 con 0x7fb64013f710 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb630001a10 con 0x7fb64013f710 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.160 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:06.161 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:06.161 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:06.161 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:06.161 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:06.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.160+0000 7fb635ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:06.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.168+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981932550 0 0) 0x7fb64006f490 con 0x7fb6400a98f0 2024-03-25T19:37:06.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.168+0000 7fb635ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.168+0000 7fb635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.168+0000 7fb635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.168+0000 7fb635ffb640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:37:06.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.168+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb624004eb0 con 0x7fb6400a98f0 2024-03-25T19:37:06.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.172+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035682056 0 0) 0x7fb6400704d0 con 0x7fb6401430c0 2024-03-25T19:37:06.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.172+0000 7fb635ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.172+0000 7fb635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.172+0000 7fb635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.172+0000 7fb635ffb640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:37:06.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.172+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64006f490 con 0x7fb6401430c0 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 179728332 0 0) 0x7fb6401df0d0 con 0x7fb64013f710 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 >> v1:172.21.15.47:6790/0 conn(0x7fb6400a98f0 legacy=0x7fb64006ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 >> v1:172.21.15.47:6789/0 conn(0x7fb6401430c0 legacy=0x7fb640138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 1 monclient: found mon.b 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6401dfcc0 con 0x7fb64013f710 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 5 monclient: authenticate success, global_id 4333 2024-03-25T19:37:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 10 monclient: _renew_subs 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64006e0e0 con 0x7fb64013f710 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 10 monclient: _renew_subs 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb640070db0 con 0x7fb64013f710 2024-03-25T19:37:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.178+0000 7fb64742a640 1 librados: init done 2024-03-25T19:37:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.179+0000 7fb64742a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.179+0000 7fb64742a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.179+0000 7fb64742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.179+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6401def30 con 0x7fb64013f710 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb630001cb0 con 0x7fb64013f710 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb635ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb6300038c0 con 0x7fb64013f710 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb635ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb635ffb640 10 monclient: dump: 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.465 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:06.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.465+0000 7fb6377fe640 4 set_mon_vals no callback set 2024-03-25T19:37:06.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.466+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fb630003ae0 con 0x7fb64013f710 2024-03-25T19:37:06.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.467+0000 7fb6451c6640 10 monclient: get_auth_request con 0x7fb62406de80 auth_method 0 2024-03-25T19:37:06.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.467+0000 7fb635ffb640 10 client.4333.objecter ms_handle_connect 0x7fb62406de80 2024-03-25T19:37:06.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fb6300546f0 con 0x7fb64013f710 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 10 client.4333.objecter ms_dispatch 0x7fb640000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb630050200 con 0x7fb64013f710 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.471+0000 7fb635ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:06.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.689+0000 7fb64742a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.689+0000 7fb64742a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.689+0000 7fb64742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:06.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.689+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6401dfe50 con 0x7fb64013f710 2024-03-25T19:37:06.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.842+0000 7fb635ffb640 1 -- 172.21.15.47:0/3848808412 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fb630052130 con 0x7fb64013f710 2024-03-25T19:37:06.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.842+0000 7fb635ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.842+0000 7fb635ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:06.843 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:06.843 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 10 librados: watch_flush enter 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 10 librados: watch_flush exit 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 10 client.4333.objecter successfully canceled tick 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 >> v1:172.21.15.142:6800/1025302268 conn(0x7fb62406de80 legacy=0x7fb624070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 10 monclient: shutdown 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 >> v1:172.21.15.142:6789/0 conn(0x7fb64013f710 legacy=0x7fb6401df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 shutdown_connections 2024-03-25T19:37:06.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 >> 172.21.15.47:0/3848808412 conn(0x7fb6400b7ea0 msgr2=0x7fb64014e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:06.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 shutdown_connections 2024-03-25T19:37:06.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 -- 172.21.15.47:0/3848808412 wait complete. 2024-03-25T19:37:06.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:06.845+0000 7fb64742a640 1 librados: shutdown 2024-03-25T19:37:06.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:13.661 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.775+0000 7f5b2c005640 10 monclient: get_monmap_and_config 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.775+0000 7f5b2c005640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.775+0000 7f5b2c005640 10 monclient: monmap: 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:13.776497+0000 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:13.776497+0000 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:13.775 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.776+0000 7f5b2c005640 1 Processor -- start 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.776+0000 7f5b2c005640 1 -- start start 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.776+0000 7f5b2c005640 10 monclient: init 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient(hunting): picked mon.noname-b con 0x7f5b24061830 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2413c9f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient(hunting): picked mon.noname-c con 0x7f5b24072af0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24071fc0 con 0x7f5b2413c9f0 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240707a0 con 0x7f5b24072af0 2024-03-25T19:37:13.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240ae9e0 con 0x7f5b24061830 2024-03-25T19:37:13.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.777+0000 7f5b2c005640 10 monclient(hunting): authenticate will time out at 5749.063651s 2024-03-25T19:37:13.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.788+0000 7f5b295a0640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5b2413c9f0 0x7f5b240723e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:59836/0 (socket says 172.21.15.47:59836) 2024-03-25T19:37:13.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.788+0000 7f5b295a0640 1 -- 172.21.15.47:0/1646848090 learned_addr learned my addr 172.21.15.47:0/1646848090 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.811+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084839655 0 0) 0x7f5b240ae9e0 con 0x7f5b24061830 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.811+0000 7f5b28d9f640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.811+0000 7f5b28d9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.811+0000 7f5b28d9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.811+0000 7f5b28d9f640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.811+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10003520 con 0x7f5b24061830 2024-03-25T19:37:13.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.823+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641681731 0 0) 0x7f5b240707a0 con 0x7f5b24072af0 2024-03-25T19:37:13.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.823+0000 7f5b28d9f640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.823+0000 7f5b28d9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.823+0000 7f5b28d9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.823+0000 7f5b28d9f640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:37:13.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.823+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240ae9e0 con 0x7f5b24072af0 2024-03-25T19:37:13.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.828+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442523910 0 0) 0x7f5b24071fc0 con 0x7f5b2413c9f0 2024-03-25T19:37:13.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.828+0000 7f5b28d9f640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.828+0000 7f5b28d9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.828+0000 7f5b28d9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.828+0000 7f5b28d9f640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:37:13.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.828+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240707a0 con 0x7f5b2413c9f0 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3477171319 0 0) 0x7f5b240ae9e0 con 0x7f5b24072af0 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 >> v1:172.21.15.47:6790/0 conn(0x7f5b24061830 legacy=0x7f5b2413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 >> v1:172.21.15.47:6789/0 conn(0x7f5b2413c9f0 legacy=0x7f5b240723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 1 monclient: found mon.noname-c 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24070cc0 con 0x7f5b24072af0 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b28d9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b2c005640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.836+0000 7f5b2c005640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.843+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b1c0019b0 con 0x7f5b24072af0 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.843+0000 7f5b28d9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.843+0000 7f5b28d9f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.843+0000 7f5b28d9f640 10 monclient: dump: 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.842 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.843+0000 7f5b2c005640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b1c001c00 con 0x7f5b24072af0 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b28d9f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b28d9f640 1 -- 172.21.15.47:0/1646848090 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b1c001f90 con 0x7f5b24072af0 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b28d9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b28d9f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b28d9f640 10 monclient: dump: 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.852 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b2c005640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:13.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.852+0000 7f5b2c005640 4 set_mon_vals no callback set 2024-03-25T19:37:13.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 10 monclient: shutdown 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 1 -- 172.21.15.47:0/1646848090 >> v1:172.21.15.142:6789/0 conn(0x7f5b24072af0 legacy=0x7f5b2414a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 1 -- 172.21.15.47:0/1646848090 shutdown_connections 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 1 -- 172.21.15.47:0/1646848090 >> 172.21.15.47:0/1646848090 conn(0x7f5b240b0ef0 msgr2=0x7f5b240b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 1 -- 172.21.15.47:0/1646848090 shutdown_connections 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.853+0000 7f5b2c005640 1 -- 172.21.15.47:0/1646848090 wait complete. 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.854+0000 7f5b2c005640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.854+0000 7f5b2c005640 10 monclient: monmap: 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.855 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 1 librados: starting msgr at 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 1 librados: starting objecter 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 1 Processor -- start 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 1 -- start start 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 1 librados: setting wanted keys 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 1 librados: calling monclient init 2024-03-25T19:37:13.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.855+0000 7f5b2c005640 10 monclient: init 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 10 monclient(hunting): picked mon.noname-c con 0x7f5b24061830 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 10 monclient(hunting): picked mon.noname-b con 0x7f5b24072af0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2413c9f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24070210 con 0x7f5b2413c9f0 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b3430 con 0x7f5b24072af0 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24133ea0 con 0x7f5b24061830 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.856+0000 7f5b2c005640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.857+0000 7f5b2c005640 10 monclient(hunting): authenticate will time out at 5749.142961s 2024-03-25T19:37:13.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.870+0000 7f5b295a0640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f5b24072af0 0x7f5b240a96c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:52068/0 (socket says 172.21.15.47:52068) 2024-03-25T19:37:13.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.870+0000 7f5b295a0640 1 -- 172.21.15.47:0/1907107437 learned_addr learned my addr 172.21.15.47:0/1907107437 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:13.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.872+0000 7f5b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f5b24061830 2024-03-25T19:37:13.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.872+0000 7f5b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.875+0000 7f5b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f5b24072af0 2024-03-25T19:37:13.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.875+0000 7f5b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.878+0000 7f5b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f5b2413c9f0 2024-03-25T19:37:13.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.878+0000 7f5b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.880+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855805587 0 0) 0x7f5b240b3430 con 0x7f5b24072af0 2024-03-25T19:37:13.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.880+0000 7f5b1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.880+0000 7f5b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.880+0000 7f5b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.880+0000 7f5b1a7fc640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:37:13.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.880+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08004ee0 con 0x7f5b24072af0 2024-03-25T19:37:13.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.881+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375904928 0 0) 0x7f5b24133ea0 con 0x7f5b24061830 2024-03-25T19:37:13.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.881+0000 7f5b1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.881+0000 7f5b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.881+0000 7f5b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.881+0000 7f5b1a7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:37:13.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.881+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240b3430 con 0x7f5b24061830 2024-03-25T19:37:13.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.892+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863584720 0 0) 0x7f5b24070210 con 0x7f5b2413c9f0 2024-03-25T19:37:13.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.892+0000 7f5b1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.892+0000 7f5b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.892+0000 7f5b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.892+0000 7f5b1a7fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:37:13.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.892+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b24133ea0 con 0x7f5b2413c9f0 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488971575 0 0) 0x7f5b08004ee0 con 0x7f5b24072af0 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b24070210 con 0x7f5b24072af0 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b14001910 con 0x7f5b24072af0 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:13.898 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:13.899 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:13.899 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:13.899 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:13.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.898+0000 7f5b1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.900+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3275551610 0 0) 0x7f5b240b3430 con 0x7f5b24061830 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.900+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b08004ee0 con 0x7f5b24061830 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.900+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b20001950 con 0x7f5b24061830 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.900+0000 7f5b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.900+0000 7f5b1a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.900+0000 7f5b1a7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:13.900 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:13.901 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:13.901 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:13.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.909+0000 7f5b29da1640 1 -- 172.21.15.47:0/1907107437 >> v1:172.21.15.47:6790/0 conn(0x7f5b24061830 legacy=0x7f5b24132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:37:13.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.909+0000 7f5b29da1640 1 -- 172.21.15.47:0/1907107437 >> v1:172.21.15.47:6790/0 conn(0x7f5b24061830 legacy=0x7f5b24132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:37:13.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.909+0000 7f5b29da1640 1 --1- 172.21.15.47:0/1907107437 >> v1:172.21.15.47:6790/0 conn(0x7f5b24061830 0x7f5b24132790 :-1 s=OPENED pgs=14819 cs=1 l=1).handle_message read tag failed 2024-03-25T19:37:13.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.909+0000 7f5b29da1640 1 --1- 172.21.15.47:0/1907107437 >> v1:172.21.15.47:6790/0 conn(0x7f5b24061830 0x7f5b24132790 :-1 s=OPENED pgs=14819 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:37:13.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.909+0000 7f5b1a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-03-25T19:37:13.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2267769782 0 0) 0x7f5b24070210 con 0x7f5b24072af0 2024-03-25T19:37:13.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 >> v1:172.21.15.47:6790/0 conn(0x7f5b24061830 legacy=0x7f5b24132790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:37:13.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 >> v1:172.21.15.47:6789/0 conn(0x7f5b2413c9f0 legacy=0x7f5b241dd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 1 monclient: found mon.b 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24132ea0 con 0x7f5b24072af0 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 10 monclient: _renew_subs 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240b38c0 con 0x7f5b24072af0 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:13.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 10 monclient: _renew_subs 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b24131d00 con 0x7f5b24072af0 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.916+0000 7f5b2c005640 1 librados: init done 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.917+0000 7f5b2c005640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.917+0000 7f5b2c005640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.917+0000 7f5b2c005640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:13.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:13.917+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b240b3b50 con 0x7f5b24072af0 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b140047e0 con 0x7f5b24072af0 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b14004a50 con 0x7f5b24072af0 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1a7fc640 10 monclient: dump: 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:14.203 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:14.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.203+0000 7f5b1bfff640 4 set_mon_vals no callback set 2024-03-25T19:37:14.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.204+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f5b14004c70 con 0x7f5b24072af0 2024-03-25T19:37:14.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.205+0000 7f5b29da1640 10 monclient: get_auth_request con 0x7f5b0806df90 auth_method 0 2024-03-25T19:37:14.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.205+0000 7f5b1a7fc640 10 client.4339.objecter ms_handle_connect 0x7f5b0806df90 2024-03-25T19:37:14.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.210+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f5b140546e0 con 0x7f5b24072af0 2024-03-25T19:37:14.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.210+0000 7f5b1a7fc640 10 client.4339.objecter ms_dispatch 0x7f5b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:14.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.210+0000 7f5b1a7fc640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:14.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.210+0000 7f5b1a7fc640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:14.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.210+0000 7f5b1a7fc640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T19:37:14.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.210+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b140501f0 con 0x7f5b24072af0 2024-03-25T19:37:14.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.211+0000 7f5b1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:14.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.211+0000 7f5b1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:14.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.474+0000 7f5b2c005640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.474+0000 7f5b2c005640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.474+0000 7f5b2c005640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:14.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.474+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b240627b0 con 0x7f5b24072af0 2024-03-25T19:37:14.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.616+0000 7f5b1a7fc640 1 -- 172.21.15.47:0/1907107437 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f5b14052120 con 0x7f5b24072af0 2024-03-25T19:37:14.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.616+0000 7f5b1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.616+0000 7f5b1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:14.616 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:14.616 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 10 librados: watch_flush enter 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 10 librados: watch_flush exit 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 10 client.4339.objecter successfully canceled tick 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 >> v1:172.21.15.142:6800/1025302268 conn(0x7f5b0806df90 legacy=0x7f5b08070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 10 monclient: shutdown 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 >> v1:172.21.15.142:6789/0 conn(0x7f5b24072af0 legacy=0x7f5b240a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:14.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 shutdown_connections 2024-03-25T19:37:14.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 >> 172.21.15.47:0/1907107437 conn(0x7f5b240b0ef0 msgr2=0x7f5b24135b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:14.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 shutdown_connections 2024-03-25T19:37:14.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 -- 172.21.15.47:0/1907107437 wait complete. 2024-03-25T19:37:14.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:14.619+0000 7f5b2c005640 1 librados: shutdown 2024-03-25T19:37:14.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:17.475 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:54.710 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.480 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:54.716 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.513 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:54.749 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.532 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:54.722 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.539 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:54.775 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.540 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:54.730 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.551 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:54.787 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.580 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:02:54.771 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:17.586 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:54.822 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:21.435 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:21.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.555+0000 7ff4d98ff640 10 monclient: get_monmap_and_config 2024-03-25T19:37:21.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.555+0000 7ff4d98ff640 10 monclient: build_initial_monmap 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.555+0000 7ff4d98ff640 10 monclient: monmap: 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:21.556159+0000 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:21.556159+0000 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:21.555 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.556+0000 7ff4d98ff640 1 Processor -- start 2024-03-25T19:37:21.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.556+0000 7ff4d98ff640 1 -- start start 2024-03-25T19:37:21.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.556+0000 7ff4d98ff640 10 monclient: init 2024-03-25T19:37:21.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.556+0000 7ff4d98ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.556+0000 7ff4d98ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.556+0000 7ff4d98ff640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d40a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d413f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d41430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d4142520 con 0x7ff4d40a98f0 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d40b28a0 con 0x7ff4d41430c0 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d40b2a30 con 0x7ff4d413f710 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:21.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.557+0000 7ff4d98ff640 10 monclient(hunting): authenticate will time out at 5756.843081s 2024-03-25T19:37:21.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.566+0000 7ff4d27fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff4d413f710 0x7ff4d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:57850/0 (socket says 172.21.15.47:57850) 2024-03-25T19:37:21.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.566+0000 7ff4d27fc640 1 -- 172.21.15.47:0/2386867818 learned_addr learned my addr 172.21.15.47:0/2386867818 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:21.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.575+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494014701 0 0) 0x7ff4d40b2a30 con 0x7ff4d413f710 2024-03-25T19:37:21.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.575+0000 7ff4d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.575+0000 7ff4d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.575+0000 7ff4d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.575+0000 7ff4d1ffb640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:37:21.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.575+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc003520 con 0x7ff4d413f710 2024-03-25T19:37:21.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.584+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687799748 0 0) 0x7ff4d40b28a0 con 0x7ff4d41430c0 2024-03-25T19:37:21.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.584+0000 7ff4d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.584+0000 7ff4d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.584+0000 7ff4d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.584+0000 7ff4d1ffb640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:37:21.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.584+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d40b2a30 con 0x7ff4d41430c0 2024-03-25T19:37:21.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.585+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334585714 0 0) 0x7ff4d4142520 con 0x7ff4d40a98f0 2024-03-25T19:37:21.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.585+0000 7ff4d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.585+0000 7ff4d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.585+0000 7ff4d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.585+0000 7ff4d1ffb640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:37:21.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.585+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d40b28a0 con 0x7ff4d40a98f0 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787840471 0 0) 0x7ff4bc003520 con 0x7ff4d413f710 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 >> v1:172.21.15.142:6789/0 conn(0x7ff4d41430c0 legacy=0x7ff4d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 >> v1:172.21.15.47:6789/0 conn(0x7ff4d40a98f0 legacy=0x7ff4d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 1 monclient: found mon.noname-b 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40b44c0 con 0x7ff4d413f710 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff4c8001690 con 0x7ff4d413f710 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d98ff640 5 monclient: authenticate success, global_id 4334 2024-03-25T19:37:21.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d98ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d1ffb640 10 monclient: dump: 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.587+0000 7ff4d98ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c80017f0 con 0x7ff4d413f710 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d1ffb640 1 -- 172.21.15.47:0/2386867818 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff4c8001c00 con 0x7ff4d413f710 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d1ffb640 10 monclient: dump: 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:21.593 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:21.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 4 set_mon_vals no callback set 2024-03-25T19:37:21.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 10 monclient: shutdown 2024-03-25T19:37:21.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:21.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 1 -- 172.21.15.47:0/2386867818 >> v1:172.21.15.47:6790/0 conn(0x7ff4d413f710 legacy=0x7ff4d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 1 -- 172.21.15.47:0/2386867818 shutdown_connections 2024-03-25T19:37:21.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 1 -- 172.21.15.47:0/2386867818 >> 172.21.15.47:0/2386867818 conn(0x7ff4d40b7ea0 msgr2=0x7ff4d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.594+0000 7ff4d98ff640 1 -- 172.21.15.47:0/2386867818 shutdown_connections 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.595+0000 7ff4d98ff640 1 -- 172.21.15.47:0/2386867818 wait complete. 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.595+0000 7ff4d98ff640 10 monclient: build_initial_monmap 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.595+0000 7ff4d98ff640 10 monclient: monmap: 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 1 librados: starting msgr at 2024-03-25T19:37:21.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 1 librados: starting objecter 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 1 Processor -- start 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 1 -- start start 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 1 librados: setting wanted keys 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 1 librados: calling monclient init 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.596+0000 7ff4d98ff640 10 monclient: init 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d40a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d413f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d41430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d4138b50 con 0x7ff4d40a98f0 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d40aba50 con 0x7ff4d41430c0 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d41d8320 con 0x7ff4d413f710 2024-03-25T19:37:21.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:21.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.597+0000 7ff4d98ff640 10 monclient(hunting): authenticate will time out at 5756.883297s 2024-03-25T19:37:21.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.606+0000 7ff4d27fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff4d413f710 0x7ff4d41d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:57866/0 (socket says 172.21.15.47:57866) 2024-03-25T19:37:21.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.606+0000 7ff4d27fc640 1 -- 172.21.15.47:0/216450700 learned_addr learned my addr 172.21.15.47:0/216450700 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:21.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.606+0000 7ff4bbfff640 10 client.?.objecter ms_handle_connect 0x7ff4d413f710 2024-03-25T19:37:21.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.606+0000 7ff4bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.619+0000 7ff4bbfff640 10 client.?.objecter ms_handle_connect 0x7ff4d40a98f0 2024-03-25T19:37:21.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.619+0000 7ff4bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.622+0000 7ff4bbfff640 10 client.?.objecter ms_handle_connect 0x7ff4d41430c0 2024-03-25T19:37:21.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.622+0000 7ff4bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.623+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594232306 0 0) 0x7ff4d41d8320 con 0x7ff4d413f710 2024-03-25T19:37:21.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.624+0000 7ff4bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.624+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.624+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.624+0000 7ff4bbfff640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.624+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c0004eb0 con 0x7ff4d413f710 2024-03-25T19:37:21.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.627+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775041114 0 0) 0x7ff4d4138b50 con 0x7ff4d40a98f0 2024-03-25T19:37:21.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.627+0000 7ff4bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.627+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.627+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.627+0000 7ff4bbfff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:37:21.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.627+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d41d8320 con 0x7ff4d40a98f0 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358095903 0 0) 0x7ff4c0004eb0 con 0x7ff4d413f710 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4d4138b50 con 0x7ff4d413f710 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff4c8001b40 con 0x7ff4d413f710 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 10 monclient(hunting): dump: 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.639 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.639+0000 7ff4bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.640+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757936609 0 0) 0x7ff4d40aba50 con 0x7ff4d41430c0 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.640+0000 7ff4bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.640+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.640+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.640+0000 7ff4bbfff640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:37:21.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.640+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c0004eb0 con 0x7ff4d41430c0 2024-03-25T19:37:21.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.644+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085537401 0 0) 0x7ff4d41d8320 con 0x7ff4d40a98f0 2024-03-25T19:37:21.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.644+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4d40aba50 con 0x7ff4d40a98f0 2024-03-25T19:37:21.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.644+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff4c4001870 con 0x7ff4d40a98f0 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.644+0000 7ff4bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.644+0000 7ff4bbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.644+0000 7ff4bbfff640 10 monclient(hunting): dump: 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:21.644 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.651+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1042322960 0 0) 0x7ff4c0004eb0 con 0x7ff4d41430c0 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.651+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4d41d8320 con 0x7ff4d41430c0 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.651+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff4cc001bd0 con 0x7ff4d41430c0 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.651+0000 7ff4bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.651+0000 7ff4bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.651+0000 7ff4bbfff640 10 monclient(hunting): dump: 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:21.651 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:21.652 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1682555826 0 0) 0x7ff4d4138b50 con 0x7ff4d413f710 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 >> v1:172.21.15.142:6789/0 conn(0x7ff4d41430c0 legacy=0x7ff4d41340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 >> v1:172.21.15.47:6789/0 conn(0x7ff4d40a98f0 legacy=0x7ff4d4138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 1 monclient: found mon.c 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d41347b0 con 0x7ff4d413f710 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 10 monclient: _renew_subs 2024-03-25T19:37:21.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d41349a0 con 0x7ff4d413f710 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 10 monclient: _renew_subs 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d4135050 con 0x7ff4d413f710 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.653+0000 7ff4d98ff640 1 librados: init done 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.654+0000 7ff4d98ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.654+0000 7ff4d98ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.654+0000 7ff4d98ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:21.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.654+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d4137f00 con 0x7ff4d413f710 2024-03-25T19:37:21.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c8004a90 con 0x7ff4d413f710 2024-03-25T19:37:21.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff4c8004da0 con 0x7ff4d413f710 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 10 monclient: dump: 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:21.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4d17fa640 4 set_mon_vals no callback set 2024-03-25T19:37:21.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.967+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ff4c801ad70 con 0x7ff4d413f710 2024-03-25T19:37:21.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.968+0000 7ff4d2ffd640 10 monclient: get_auth_request con 0x7ff4c006e110 auth_method 0 2024-03-25T19:37:21.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.969+0000 7ff4bbfff640 10 client.4337.objecter ms_handle_connect 0x7ff4c006e110 2024-03-25T19:37:21.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7ff4c80548c0 con 0x7ff4d413f710 2024-03-25T19:37:21.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 10 client.4337.objecter ms_dispatch 0x7ff4d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:21.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:21.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:21.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T19:37:21.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4c80503d0 con 0x7ff4d413f710 2024-03-25T19:37:21.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:21.974+0000 7ff4bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:22.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.226+0000 7ff4d98ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:22.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.226+0000 7ff4d98ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:22.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.226+0000 7ff4d98ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:22.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.226+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4d40627b0 con 0x7ff4d413f710 2024-03-25T19:37:22.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.357+0000 7ff4bbfff640 1 -- 172.21.15.47:0/216450700 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7ff4c8052300 con 0x7ff4d413f710 2024-03-25T19:37:22.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.357+0000 7ff4bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:22.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.357+0000 7ff4bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:22.357 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:22.358 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:22.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 10 librados: watch_flush enter 2024-03-25T19:37:22.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 10 librados: watch_flush exit 2024-03-25T19:37:22.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T19:37:22.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 10 client.4337.objecter successfully canceled tick 2024-03-25T19:37:22.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 >> v1:172.21.15.142:6800/1025302268 conn(0x7ff4c006e110 legacy=0x7ff4c00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 10 monclient: shutdown 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 >> v1:172.21.15.47:6790/0 conn(0x7ff4d413f710 legacy=0x7ff4d41d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 shutdown_connections 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.359+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 >> 172.21.15.47:0/216450700 conn(0x7ff4d40b7ea0 msgr2=0x7ff4d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.360+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 shutdown_connections 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.360+0000 7ff4d98ff640 1 -- 172.21.15.47:0/216450700 wait complete. 2024-03-25T19:37:22.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:22.360+0000 7ff4d98ff640 1 librados: shutdown 2024-03-25T19:37:22.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:29.178 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.298+0000 7f34b8693640 10 monclient: get_monmap_and_config 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.298+0000 7f34b8693640 10 monclient: build_initial_monmap 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.298+0000 7f34b8693640 10 monclient: monmap: 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:29.299942+0000 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:29.299942+0000 2024-03-25T19:37:29.298 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.298+0000 7f34b8693640 1 Processor -- start 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.299+0000 7f34b8693640 1 -- start start 2024-03-25T19:37:29.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.299+0000 7f34b8693640 10 monclient: init 2024-03-25T19:37:29.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient(hunting): picked mon.noname-c con 0x7f34b006c380 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient(hunting): picked mon.noname-b con 0x7f34b00b12d0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient(hunting): picked mon.noname-a con 0x7f34b00aedf0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00b16f0 con 0x7f34b00aedf0 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00adfe0 con 0x7f34b006c380 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00add00 con 0x7f34b00b12d0 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:29.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.300+0000 7f34b8693640 10 monclient(hunting): authenticate will time out at 5764.587398s 2024-03-25T19:37:29.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.318+0000 7f34b6c30640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f34b00aedf0 0x7f34b014e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52118/0 (socket says 172.21.15.47:52118) 2024-03-25T19:37:29.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.318+0000 7f34b6c30640 1 -- 172.21.15.47:0/3426716843 learned_addr learned my addr 172.21.15.47:0/3426716843 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:29.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.326+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831880780 0 0) 0x7f34b00b16f0 con 0x7f34b00aedf0 2024-03-25T19:37:29.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.326+0000 7f34b542d640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.326+0000 7f34b542d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.326+0000 7f34b542d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.326+0000 7f34b542d640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:37:29.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.326+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f349c003520 con 0x7f34b00aedf0 2024-03-25T19:37:29.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.328+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648166870 0 0) 0x7f34b00adfe0 con 0x7f34b006c380 2024-03-25T19:37:29.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.328+0000 7f34b542d640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.328+0000 7f34b542d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.328+0000 7f34b542d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.328+0000 7f34b542d640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:37:29.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.328+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b00b16f0 con 0x7f34b006c380 2024-03-25T19:37:29.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.331+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235037290 0 0) 0x7f34b00add00 con 0x7f34b00b12d0 2024-03-25T19:37:29.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.331+0000 7f34b542d640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.331+0000 7f34b542d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.331+0000 7f34b542d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.331+0000 7f34b542d640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:37:29.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.331+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b00adfe0 con 0x7f34b00b12d0 2024-03-25T19:37:29.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466197592 0 0) 0x7f349c003520 con 0x7f34b00aedf0 2024-03-25T19:37:29.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 >> v1:172.21.15.47:6790/0 conn(0x7f34b00b12d0 legacy=0x7f34b00ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 >> v1:172.21.15.142:6789/0 conn(0x7f34b006c380 legacy=0x7f34b00b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: _finish_hunting 0 2024-03-25T19:37:29.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 1 monclient: found mon.noname-a 2024-03-25T19:37:29.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b00ae2c0 con 0x7f34b00aedf0 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: _finish_auth 0 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: _check_auth_tickets 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f34a8001a10 con 0x7f34b00aedf0 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b8693640 5 monclient: authenticate success, global_id 4356 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b8693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b542d640 10 monclient: dump: 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.337 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.338 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:29.338 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:29.338 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:29.338 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:29.338 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.335+0000 7f34b8693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:29.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34a8001b70 con 0x7f34b00aedf0 2024-03-25T19:37:29.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b542d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:29.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b542d640 1 -- 172.21.15.47:0/3426716843 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f34a8001f00 con 0x7f34b00aedf0 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b8693640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b8693640 4 set_mon_vals no callback set 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b542d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b542d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.349+0000 7f34b542d640 10 monclient: dump: 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:29.350 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:29.351 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:29.351 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:29.351 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 10 monclient: shutdown 2024-03-25T19:37:29.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:29.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 1 -- 172.21.15.47:0/3426716843 >> v1:172.21.15.47:6789/0 conn(0x7f34b00aedf0 legacy=0x7f34b014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 1 -- 172.21.15.47:0/3426716843 shutdown_connections 2024-03-25T19:37:29.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 1 -- 172.21.15.47:0/3426716843 >> 172.21.15.47:0/3426716843 conn(0x7f34b00b5d10 msgr2=0x7f34b00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 1 -- 172.21.15.47:0/3426716843 shutdown_connections 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.350+0000 7f34b8693640 1 -- 172.21.15.47:0/3426716843 wait complete. 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 10 monclient: build_initial_monmap 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 10 monclient: monmap: 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:29.354 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 1 librados: starting msgr at 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 1 librados: starting objecter 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 1 Processor -- start 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 1 -- start start 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 1 librados: setting wanted keys 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 1 librados: calling monclient init 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.351+0000 7f34b8693640 10 monclient: init 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.352+0000 7f34b8693640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.352+0000 7f34b8693640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.352+0000 7f34b8693640 10 monclient(hunting): picked mon.noname-c con 0x7f34b006c380 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:29.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.352+0000 7f34b8693640 10 monclient(hunting): picked mon.noname-b con 0x7f34b00aedf0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.352+0000 7f34b8693640 10 monclient(hunting): picked mon.noname-a con 0x7f34b00b12d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.353+0000 7f34b8693640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b006c8e0 con 0x7f34b00b12d0 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.353+0000 7f34b8693640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b01c27a0 con 0x7f34b00aedf0 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.353+0000 7f34b8693640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b01c4670 con 0x7f34b006c380 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.353+0000 7f34b8693640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.353+0000 7f34b8693640 10 monclient(hunting): authenticate will time out at 5764.640034s 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.356+0000 7f34b5c2e640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f34b00aedf0 0x7f34b01da9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:51702/0 (socket says 172.21.15.47:51702) 2024-03-25T19:37:29.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.356+0000 7f34b5c2e640 1 -- 172.21.15.47:0/437530042 learned_addr learned my addr 172.21.15.47:0/437530042 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:29.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.359+0000 7f34a6ffd640 10 client.?.objecter ms_handle_connect 0x7f34b00aedf0 2024-03-25T19:37:29.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.359+0000 7f34a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:29.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.369+0000 7f34a6ffd640 10 client.?.objecter ms_handle_connect 0x7f34b006c380 2024-03-25T19:37:29.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.369+0000 7f34a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:29.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.369+0000 7f34a6ffd640 10 client.?.objecter ms_handle_connect 0x7f34b00b12d0 2024-03-25T19:37:29.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.369+0000 7f34a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:29.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.373+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634970484 0 0) 0x7f34b01c27a0 con 0x7f34b00aedf0 2024-03-25T19:37:29.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.373+0000 7f34a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.373+0000 7f34a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.373+0000 7f34a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.373+0000 7f34a6ffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:37:29.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.373+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494004eb0 con 0x7f34b00aedf0 2024-03-25T19:37:29.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.376+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813595849 0 0) 0x7f34b006c8e0 con 0x7f34b00b12d0 2024-03-25T19:37:29.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.376+0000 7f34a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.376+0000 7f34a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.376+0000 7f34a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.376+0000 7f34a6ffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:37:29.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.376+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b01c27a0 con 0x7f34b00b12d0 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311619986 0 0) 0x7f3494004eb0 con 0x7f34b00aedf0 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b006c8e0 con 0x7f34b00aedf0 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f34a0001970 con 0x7f34b00aedf0 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.385 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730775050 0 0) 0x7f34b01c4670 con 0x7f34b006c380 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:37:29.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.384+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494004eb0 con 0x7f34b006c380 2024-03-25T19:37:29.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35777252 0 0) 0x7f3494004eb0 con 0x7f34b006c380 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b01c4670 con 0x7f34b006c380 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f34ac001930 con 0x7f34b006c380 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 241255977 0 0) 0x7f34b006c8e0 con 0x7f34b00aedf0 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 >> v1:172.21.15.47:6790/0 conn(0x7f34b006c380 legacy=0x7f34b01c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 >> v1:172.21.15.47:6789/0 conn(0x7f34b00b12d0 legacy=0x7f34b01c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 monclient: found mon.b 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b01db100 con 0x7f34b00aedf0 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 10 monclient: _renew_subs 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:29.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b01c1830 con 0x7f34b00aedf0 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 10 monclient: _renew_subs 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b01c10d0 con 0x7f34b00aedf0 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.397+0000 7f34b8693640 1 librados: init done 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.399+0000 7f34b8693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.399+0000 7f34b8693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.399+0000 7f34b8693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:29.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.399+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b00627b0 con 0x7f34b00aedf0 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34a00035f0 con 0x7f34b00aedf0 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f34a0003980 con 0x7f34b00aedf0 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34a6ffd640 10 monclient: dump: 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:29.696 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.695+0000 7f34b4c2c640 4 set_mon_vals no callback set 2024-03-25T19:37:29.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.696+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f34a0003ba0 con 0x7f34b00aedf0 2024-03-25T19:37:29.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.696+0000 7f34b642f640 10 monclient: get_auth_request con 0x7f349406dfb0 auth_method 0 2024-03-25T19:37:29.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.696+0000 7f34a6ffd640 10 client.4351.objecter ms_handle_connect 0x7f349406dfb0 2024-03-25T19:37:29.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.704+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f34a0054760 con 0x7f34b00aedf0 2024-03-25T19:37:29.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.704+0000 7f34a6ffd640 10 client.4351.objecter ms_dispatch 0x7f34b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:29.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.704+0000 7f34a6ffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:29.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.704+0000 7f34a6ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:29.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.704+0000 7f34a6ffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:37:29.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.706+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34a0050270 con 0x7f34b00aedf0 2024-03-25T19:37:29.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.706+0000 7f34a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.706+0000 7f34a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:29.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.962+0000 7f34b8693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.962+0000 7f34b8693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.962+0000 7f34b8693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:29.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.962+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34b0069f40 con 0x7f34b00aedf0 2024-03-25T19:37:29.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.993+0000 7f34a6ffd640 1 -- 172.21.15.47:0/437530042 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f34a00521a0 con 0x7f34b00aedf0 2024-03-25T19:37:29.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.993+0000 7f34a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.993+0000 7f34a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:29.995 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:29.995 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:29.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 10 librados: watch_flush enter 2024-03-25T19:37:29.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 10 librados: watch_flush exit 2024-03-25T19:37:29.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:37:29.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:37:29.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 >> v1:172.21.15.142:6800/1025302268 conn(0x7f349406dfb0 legacy=0x7f3494070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 10 monclient: shutdown 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 >> v1:172.21.15.142:6789/0 conn(0x7f34b00aedf0 legacy=0x7f34b01da9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 shutdown_connections 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 >> 172.21.15.47:0/437530042 conn(0x7f34b00b5d10 msgr2=0x7f34b00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 shutdown_connections 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 -- 172.21.15.47:0/437530042 wait complete. 2024-03-25T19:37:29.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:29.996+0000 7f34b8693640 1 librados: shutdown 2024-03-25T19:37:30.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:36.814 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:36.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.932+0000 7fdbad0c8640 10 monclient: get_monmap_and_config 2024-03-25T19:37:36.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.932+0000 7fdbad0c8640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.932+0000 7fdbad0c8640 10 monclient: monmap: 2024-03-25T19:37:36.931 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:36.931 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:36.931 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:36.932846+0000 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:36.932846+0000 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:36.932 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:36.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.932+0000 7fdbad0c8640 1 Processor -- start 2024-03-25T19:37:36.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.932+0000 7fdbad0c8640 1 -- start start 2024-03-25T19:37:36.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.932+0000 7fdbad0c8640 10 monclient: init 2024-03-25T19:37:36.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:36.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdba80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdba813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdba81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba8142520 con 0x7fdba813f710 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba80b28a0 con 0x7fdba80a98f0 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba80b2a30 con 0x7fdba81430c0 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.933+0000 7fdbad0c8640 10 monclient(hunting): authenticate will time out at 5772.219506s 2024-03-25T19:37:36.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.938+0000 7fdba659c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdba813f710 0x7fdba81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57148/0 (socket says 172.21.15.47:57148) 2024-03-25T19:37:36.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.938+0000 7fdba659c640 1 -- 172.21.15.47:0/55641129 learned_addr learned my addr 172.21.15.47:0/55641129 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:36.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.947+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473641020 0 0) 0x7fdba8142520 con 0x7fdba813f710 2024-03-25T19:37:36.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.947+0000 7fdba5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.947+0000 7fdba5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.947+0000 7fdba5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.947+0000 7fdba5d9b640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:37:36.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.947+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90003520 con 0x7fdba813f710 2024-03-25T19:37:36.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.955+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181064251 0 0) 0x7fdba80b28a0 con 0x7fdba80a98f0 2024-03-25T19:37:36.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.955+0000 7fdba5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.955+0000 7fdba5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.955+0000 7fdba5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.955+0000 7fdba5d9b640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:37:36.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.955+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba8142520 con 0x7fdba80a98f0 2024-03-25T19:37:36.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1886959057 0 0) 0x7fdb90003520 con 0x7fdba813f710 2024-03-25T19:37:36.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 >> v1:172.21.15.47:6790/0 conn(0x7fdba81430c0 legacy=0x7fdba814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 >> v1:172.21.15.142:6789/0 conn(0x7fdba80a98f0 legacy=0x7fdba80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 1 monclient: found mon.noname-a 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba80b44c0 con 0x7fdba813f710 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdb9c001690 con 0x7fdba813f710 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdba5d9b640 10 monclient: dump: 2024-03-25T19:37:36.957 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdbad0c8640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:37:36.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.957+0000 7fdbad0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb9c0017f0 con 0x7fdba813f710 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdba5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdba5d9b640 1 -- 172.21.15.47:0/55641129 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdb9c001c00 con 0x7fdba813f710 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdba5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdba5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdba5d9b640 10 monclient: dump: 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.993 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:36.994 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:36.994 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:36.994 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:36.994 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:36.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdbad0c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:36.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.993+0000 7fdbad0c8640 4 set_mon_vals no callback set 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 10 monclient: shutdown 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 1 -- 172.21.15.47:0/55641129 >> v1:172.21.15.47:6789/0 conn(0x7fdba813f710 legacy=0x7fdba81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 1 -- 172.21.15.47:0/55641129 shutdown_connections 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 1 -- 172.21.15.47:0/55641129 >> 172.21.15.47:0/55641129 conn(0x7fdba80b7ea0 msgr2=0x7fdba80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 1 -- 172.21.15.47:0/55641129 shutdown_connections 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.994+0000 7fdbad0c8640 1 -- 172.21.15.47:0/55641129 wait complete. 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.995+0000 7fdbad0c8640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.995+0000 7fdbad0c8640 10 monclient: monmap: 2024-03-25T19:37:36.996 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.995+0000 7fdbad0c8640 1 librados: starting msgr at 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.995+0000 7fdbad0c8640 1 librados: starting objecter 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 1 Processor -- start 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 1 -- start start 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 1 librados: setting wanted keys 2024-03-25T19:37:36.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 1 librados: calling monclient init 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 10 monclient: init 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdba80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdba813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.996+0000 7fdbad0c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdba81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.997+0000 7fdbad0c8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba8139520 con 0x7fdba80a98f0 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.997+0000 7fdbad0c8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba8139800 con 0x7fdba81430c0 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.997+0000 7fdbad0c8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba81d4bc0 con 0x7fdba813f710 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.997+0000 7fdbad0c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:36.997+0000 7fdbad0c8640 10 monclient(hunting): authenticate will time out at 5772.283027s 2024-03-25T19:37:37.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.007+0000 7fdba759e640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdba81430c0 0x7fdba81e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:56528/0 (socket says 172.21.15.47:56528) 2024-03-25T19:37:37.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.007+0000 7fdba759e640 1 -- 172.21.15.47:0/917649391 learned_addr learned my addr 172.21.15.47:0/917649391 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:37.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.009+0000 7fdb8f7fe640 10 client.?.objecter ms_handle_connect 0x7fdba80a98f0 2024-03-25T19:37:37.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.009+0000 7fdb8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:37.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.010+0000 7fdb8f7fe640 10 client.?.objecter ms_handle_connect 0x7fdba81430c0 2024-03-25T19:37:37.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.010+0000 7fdb8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:37.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.017+0000 7fdb8f7fe640 10 client.?.objecter ms_handle_connect 0x7fdba813f710 2024-03-25T19:37:37.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.017+0000 7fdb8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:37.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.026+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268317222 0 0) 0x7fdba8139520 con 0x7fdba80a98f0 2024-03-25T19:37:37.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.026+0000 7fdb8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:37.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.026+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:37.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.026+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:37.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.026+0000 7fdb8f7fe640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:37:37.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.026+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb94004eb0 con 0x7fdba80a98f0 2024-03-25T19:37:37.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.027+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521918832 0 0) 0x7fdba8139800 con 0x7fdba81430c0 2024-03-25T19:37:37.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.027+0000 7fdb8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:37.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.027+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:37.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.027+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:37.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.027+0000 7fdb8f7fe640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:37:37.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.027+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba8139520 con 0x7fdba81430c0 2024-03-25T19:37:37.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.028+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513954660 0 0) 0x7fdba81d4bc0 con 0x7fdba813f710 2024-03-25T19:37:37.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.028+0000 7fdb8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:37.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.028+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:37.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.028+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:37.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.028+0000 7fdb8f7fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:37:37.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.028+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba8139800 con 0x7fdba813f710 2024-03-25T19:37:37.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733125309 0 0) 0x7fdba8139520 con 0x7fdba81430c0 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdba81d4bc0 con 0x7fdba81430c0 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdba0001a50 con 0x7fdba81430c0 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:37.036 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3417326269 0 0) 0x7fdb94004eb0 con 0x7fdba80a98f0 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdba8139520 con 0x7fdba80a98f0 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdb98001930 con 0x7fdba80a98f0 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.036+0000 7fdb8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:37.037 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:37.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.039+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369739151 0 0) 0x7fdba8139800 con 0x7fdba813f710 2024-03-25T19:37:37.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.039+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb94004eb0 con 0x7fdba813f710 2024-03-25T19:37:37.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.039+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdb9c001b00 con 0x7fdba813f710 2024-03-25T19:37:37.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.039+0000 7fdb8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.039+0000 7fdb8f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.039+0000 7fdb8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:37.039 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 959906739 0 0) 0x7fdba8139520 con 0x7fdba80a98f0 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 >> v1:172.21.15.47:6790/0 conn(0x7fdba813f710 legacy=0x7fdba81d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 >> v1:172.21.15.142:6789/0 conn(0x7fdba81430c0 legacy=0x7fdba81e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 1 monclient: found mon.a 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:37.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba80b0ca0 con 0x7fdba80a98f0 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdb8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 5 monclient: authenticate success, global_id 4365 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 10 monclient: _renew_subs 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba80b0f50 con 0x7fdba80a98f0 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 10 monclient: _renew_subs 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:37.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba80b07f0 con 0x7fdba80a98f0 2024-03-25T19:37:37.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.044+0000 7fdbad0c8640 1 librados: init done 2024-03-25T19:37:37.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.045+0000 7fdbad0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:37.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.046+0000 7fdbad0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:37.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.046+0000 7fdbad0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:37.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.046+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba8074cc0 con 0x7fdba80a98f0 2024-03-25T19:37:37.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.363+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb980035f0 con 0x7fdba80a98f0 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdb8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdb98003900 con 0x7fdba80a98f0 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdb8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdb8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdb8f7fe640 10 monclient: dump: 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:37.364 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:37.365 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:37.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdba559a640 4 set_mon_vals no callback set 2024-03-25T19:37:37.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.364+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fdb98019910 con 0x7fdba80a98f0 2024-03-25T19:37:37.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.365+0000 7fdba659c640 10 monclient: get_auth_request con 0x7fdb9406e110 auth_method 0 2024-03-25T19:37:37.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.365+0000 7fdb8f7fe640 10 client.4365.objecter ms_handle_connect 0x7fdb9406e110 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fdb98053620 con 0x7fdba80a98f0 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 10 client.4365.objecter ms_dispatch 0x7fdba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb9804ef60 con 0x7fdba80a98f0 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:37.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.371+0000 7fdb8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:37.498 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:14.733 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.549 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:14.784 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.596 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:14.832 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.601+0000 7fdbad0c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.601+0000 7fdbad0c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.601+0000 7fdbad0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:37.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.601+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba81d4d50 con 0x7fdba80a98f0 2024-03-25T19:37:37.607 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:14.843 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.613 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:14.849 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.618 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:14.808 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.624 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:14.814 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.654 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:14.889 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.659 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:14.849 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:37.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.660+0000 7fdb8f7fe640 1 -- 172.21.15.47:0/917649391 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fdb98051060 con 0x7fdba80a98f0 2024-03-25T19:37:37.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.660+0000 7fdb8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.660+0000 7fdb8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:37.660 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:37.660 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 10 librados: watch_flush enter 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 10 librados: watch_flush exit 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 10 client.4365.objecter successfully canceled tick 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 >> v1:172.21.15.142:6800/1025302268 conn(0x7fdb9406e110 legacy=0x7fdb940705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 10 monclient: shutdown 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 >> v1:172.21.15.47:6789/0 conn(0x7fdba80a98f0 legacy=0x7fdba80aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 shutdown_connections 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 >> 172.21.15.47:0/917649391 conn(0x7fdba80b7ea0 msgr2=0x7fdba8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 shutdown_connections 2024-03-25T19:37:37.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 -- 172.21.15.47:0/917649391 wait complete. 2024-03-25T19:37:37.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:37.663+0000 7fdbad0c8640 1 librados: shutdown 2024-03-25T19:37:37.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:44.481 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:44.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.598+0000 7fb58b4f1640 10 monclient: get_monmap_and_config 2024-03-25T19:37:44.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.598+0000 7fb58b4f1640 10 monclient: build_initial_monmap 2024-03-25T19:37:44.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.598+0000 7fb58b4f1640 10 monclient: monmap: 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:44.599152+0000 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:44.599152+0000 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:44.598 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 1 Processor -- start 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 1 -- start start 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 10 monclient: init 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb5840b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.599+0000 7fb58b4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fb5840b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.600+0000 7fb58b4f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb58414be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.600+0000 7fb58b4f1640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5840af940 con 0x7fb5840b1730 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.600+0000 7fb58b4f1640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5840b0a70 con 0x7fb5840b4d60 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.600+0000 7fb58b4f1640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5840b0d50 con 0x7fb58414be90 2024-03-25T19:37:44.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.600+0000 7fb58b4f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:44.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.600+0000 7fb58b4f1640 10 monclient(hunting): authenticate will time out at 5779.886082s 2024-03-25T19:37:44.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.609+0000 7fb589a8e640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb58414be90 0x7fb58414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44768/0 (socket says 172.21.15.47:44768) 2024-03-25T19:37:44.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.609+0000 7fb589a8e640 1 -- 172.21.15.47:0/413529372 learned_addr learned my addr 172.21.15.47:0/413529372 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:44.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.621+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057903123 0 0) 0x7fb5840af940 con 0x7fb5840b1730 2024-03-25T19:37:44.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.621+0000 7fb57bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.621+0000 7fb57bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.621+0000 7fb57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.621+0000 7fb57bfff640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:37:44.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.621+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb570003520 con 0x7fb5840b1730 2024-03-25T19:37:44.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.624+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974480602 0 0) 0x7fb5840b0a70 con 0x7fb5840b4d60 2024-03-25T19:37:44.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.624+0000 7fb57bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.624+0000 7fb57bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.624+0000 7fb57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.624+0000 7fb57bfff640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:37:44.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.624+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5840af940 con 0x7fb5840b4d60 2024-03-25T19:37:44.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.627+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691439936 0 0) 0x7fb5840b0d50 con 0x7fb58414be90 2024-03-25T19:37:44.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.627+0000 7fb57bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.627+0000 7fb57bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.627+0000 7fb57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.627+0000 7fb57bfff640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:37:44.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.627+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5840b0a70 con 0x7fb58414be90 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330951806 0 0) 0x7fb570003520 con 0x7fb5840b1730 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 >> v1:172.21.15.47:6790/0 conn(0x7fb58414be90 legacy=0x7fb58414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 >> v1:172.21.15.142:6789/0 conn(0x7fb5840b4d60 legacy=0x7fb58413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 1 monclient: found mon.noname-a 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5840b1030 con 0x7fb5840b1730 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: _finish_auth 0 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:44.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb574001690 con 0x7fb5840b1730 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb58b4f1640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb57bfff640 10 monclient: dump: 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:44.629 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.628+0000 7fb58b4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:44.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5740017f0 con 0x7fb5840b1730 2024-03-25T19:37:44.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb57bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:44.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb57bfff640 1 -- 172.21.15.47:0/413529372 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb574001c00 con 0x7fb5840b1730 2024-03-25T19:37:44.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb57bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb57bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:44.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb57bfff640 10 monclient: dump: 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb58b4f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:44.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb58b4f1640 4 set_mon_vals no callback set 2024-03-25T19:37:44.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb58b4f1640 10 monclient: shutdown 2024-03-25T19:37:44.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb58b4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.638+0000 7fb58b4f1640 1 -- 172.21.15.47:0/413529372 >> v1:172.21.15.47:6789/0 conn(0x7fb5840b1730 legacy=0x7fb5840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.639+0000 7fb58b4f1640 1 -- 172.21.15.47:0/413529372 shutdown_connections 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.639+0000 7fb58b4f1640 1 -- 172.21.15.47:0/413529372 >> 172.21.15.47:0/413529372 conn(0x7fb5840617a0 msgr2=0x7fb584061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.639+0000 7fb58b4f1640 1 -- 172.21.15.47:0/413529372 shutdown_connections 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.639+0000 7fb58b4f1640 1 -- 172.21.15.47:0/413529372 wait complete. 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 10 monclient: build_initial_monmap 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 10 monclient: monmap: 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:44.641 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 1 librados: starting msgr at 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 1 librados: starting objecter 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 1 Processor -- start 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 1 -- start start 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 1 librados: setting wanted keys 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 1 librados: calling monclient init 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.640+0000 7fb58b4f1640 10 monclient: init 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb5840b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb5840b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fb58414be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58413a510 con 0x7fb58414be90 2024-03-25T19:37:44.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58413a7f0 con 0x7fb5840b1730 2024-03-25T19:37:44.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5841354a0 con 0x7fb5840b4d60 2024-03-25T19:37:44.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:44.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.641+0000 7fb58b4f1640 10 monclient(hunting): authenticate will time out at 5779.927474s 2024-03-25T19:37:44.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.653+0000 7fb58928d640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb5840b1730 0x7fb584142890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:35650/0 (socket says 172.21.15.47:35650) 2024-03-25T19:37:44.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.653+0000 7fb58928d640 1 -- 172.21.15.47:0/2266693073 learned_addr learned my addr 172.21.15.47:0/2266693073 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:44.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.655+0000 7fb579ffb640 10 client.?.objecter ms_handle_connect 0x7fb5840b1730 2024-03-25T19:37:44.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.655+0000 7fb579ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.657+0000 7fb579ffb640 10 client.?.objecter ms_handle_connect 0x7fb58414be90 2024-03-25T19:37:44.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.657+0000 7fb579ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.659+0000 7fb579ffb640 10 client.?.objecter ms_handle_connect 0x7fb5840b4d60 2024-03-25T19:37:44.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.659+0000 7fb579ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.666+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201932963 0 0) 0x7fb5841354a0 con 0x7fb5840b4d60 2024-03-25T19:37:44.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.666+0000 7fb579ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.666+0000 7fb579ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.666+0000 7fb579ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.666+0000 7fb579ffb640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:37:44.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.666+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb568004eb0 con 0x7fb5840b4d60 2024-03-25T19:37:44.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.669+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942240566 0 0) 0x7fb58413a7f0 con 0x7fb5840b1730 2024-03-25T19:37:44.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.669+0000 7fb579ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.669+0000 7fb579ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.669+0000 7fb579ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.669+0000 7fb579ffb640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:37:44.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.669+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5841354a0 con 0x7fb5840b1730 2024-03-25T19:37:44.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.672+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721876616 0 0) 0x7fb58413a510 con 0x7fb58414be90 2024-03-25T19:37:44.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.672+0000 7fb579ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.672+0000 7fb579ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.672+0000 7fb579ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.672+0000 7fb579ffb640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:37:44.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.672+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58413a7f0 con 0x7fb58414be90 2024-03-25T19:37:44.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442161073 0 0) 0x7fb568004eb0 con 0x7fb5840b4d60 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb58413a510 con 0x7fb5840b4d60 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb574001b00 con 0x7fb5840b4d60 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:44.677 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:44.678 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.677+0000 7fb579ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:44.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.685+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1495422616 0 0) 0x7fb58413a7f0 con 0x7fb58414be90 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.685+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb568004eb0 con 0x7fb58414be90 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.685+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb57c001bd0 con 0x7fb58414be90 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.685+0000 7fb579ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.685+0000 7fb579ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.685+0000 7fb579ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:44.685 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:44.686 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.690+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3791662849 0 0) 0x7fb5841354a0 con 0x7fb5840b1730 2024-03-25T19:37:44.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.690+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb58413a7f0 con 0x7fb5840b1730 2024-03-25T19:37:44.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.690+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb5800019b0 con 0x7fb5840b1730 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.690+0000 7fb579ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.690+0000 7fb579ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.690+0000 7fb579ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:44.690 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 172773238 0 0) 0x7fb58413a510 con 0x7fb5840b4d60 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 >> v1:172.21.15.142:6789/0 conn(0x7fb5840b1730 legacy=0x7fb584142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 >> v1:172.21.15.47:6789/0 conn(0x7fb58414be90 legacy=0x7fb584134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 1 monclient: found mon.c 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:44.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb584143210 con 0x7fb5840b4d60 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb579ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 5 monclient: authenticate success, global_id 4352 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 10 monclient: _renew_subs 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58413bf00 con 0x7fb5840b4d60 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 10 monclient: _renew_subs 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:44.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5841dff40 con 0x7fb5840b4d60 2024-03-25T19:37:44.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.691+0000 7fb58b4f1640 1 librados: init done 2024-03-25T19:37:44.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.692+0000 7fb58b4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.692+0000 7fb58b4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.692+0000 7fb58b4f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:44.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:44.692+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb584135630 con 0x7fb5840b4d60 2024-03-25T19:37:45.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb574001220 con 0x7fb5840b4d60 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb579ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb574003a40 con 0x7fb5840b4d60 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb579ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb579ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb579ffb640 10 monclient: dump: 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:45.012 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:45.013 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:45.013 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:45.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.012+0000 7fb57b7fe640 4 set_mon_vals no callback set 2024-03-25T19:37:45.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.013+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fb574019a10 con 0x7fb5840b4d60 2024-03-25T19:37:45.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.014+0000 7fb58928d640 10 monclient: get_auth_request con 0x7fb56806e0b0 auth_method 0 2024-03-25T19:37:45.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.014+0000 7fb579ffb640 10 client.4352.objecter ms_handle_connect 0x7fb56806e0b0 2024-03-25T19:37:45.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fb574053680 con 0x7fb5840b4d60 2024-03-25T19:37:45.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 10 client.4352.objecter ms_dispatch 0x7fb584000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:45.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:45.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:45.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-25T19:37:45.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb57404efc0 con 0x7fb5840b4d60 2024-03-25T19:37:45.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:45.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.020+0000 7fb579ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.278+0000 7fb58b4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.278+0000 7fb58b4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.278+0000 7fb58b4f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:37:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.278+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5841433a0 con 0x7fb5840b4d60 2024-03-25T19:37:45.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.305+0000 7fb579ffb640 1 -- 172.21.15.47:0/2266693073 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fb5740510c0 con 0x7fb5840b4d60 2024-03-25T19:37:45.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.305+0000 7fb579ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:45.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.305+0000 7fb579ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:45.305 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:45.305 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 10 librados: watch_flush enter 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 10 librados: watch_flush exit 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 10 client.4352.objecter successfully canceled tick 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 >> v1:172.21.15.142:6800/1025302268 conn(0x7fb56806e0b0 legacy=0x7fb568070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 10 monclient: shutdown 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 >> v1:172.21.15.47:6790/0 conn(0x7fb5840b4d60 legacy=0x7fb5841df650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 shutdown_connections 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 >> 172.21.15.47:0/2266693073 conn(0x7fb5840617a0 msgr2=0x7fb5840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 shutdown_connections 2024-03-25T19:37:45.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 -- 172.21.15.47:0/2266693073 wait complete. 2024-03-25T19:37:45.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:45.308+0000 7fb58b4f1640 1 librados: shutdown 2024-03-25T19:37:45.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:52.126 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.242+0000 7f06fe75d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.242+0000 7f06fe75d640 10 monclient: build_initial_monmap 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.242+0000 7f06fe75d640 10 monclient: monmap: 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:52.243255+0000 2024-03-25T19:37:52.242 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:52.243255+0000 2024-03-25T19:37:52.243 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:52.243 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:52.243 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:52.243 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:52.243 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:52.243 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.243+0000 7f06fe75d640 1 Processor -- start 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.243+0000 7f06fe75d640 1 -- start start 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.243+0000 7f06fe75d640 10 monclient: init 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient(hunting): picked mon.noname-c con 0x7f06f80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient(hunting): picked mon.noname-b con 0x7f06f813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient(hunting): picked mon.noname-a con 0x7f06f81430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:52.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f8142520 con 0x7f06f81430c0 2024-03-25T19:37:52.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b28a0 con 0x7f06f80a98f0 2024-03-25T19:37:52.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b2a30 con 0x7f06f813f710 2024-03-25T19:37:52.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:52.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.244+0000 7f06fe75d640 10 monclient(hunting): authenticate will time out at 5787.530542s 2024-03-25T19:37:52.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.254+0000 7f06f7fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f06f80a98f0 0x7f06f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:51936/0 (socket says 172.21.15.47:51936) 2024-03-25T19:37:52.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.254+0000 7f06f7fff640 1 -- 172.21.15.47:0/2607483783 learned_addr learned my addr 172.21.15.47:0/2607483783 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:52.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.269+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518765524 0 0) 0x7f06f80b28a0 con 0x7f06f80a98f0 2024-03-25T19:37:52.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.269+0000 7f06f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.269+0000 7f06f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.269+0000 7f06f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.269+0000 7f06f6ffd640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:37:52.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.269+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0003520 con 0x7f06f80a98f0 2024-03-25T19:37:52.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.288+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452774544 0 0) 0x7f06e0003520 con 0x7f06f80a98f0 2024-03-25T19:37:52.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 >> v1:172.21.15.47:6790/0 conn(0x7f06f813f710 legacy=0x7f06f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 >> v1:172.21.15.47:6789/0 conn(0x7f06f81430c0 legacy=0x7f06f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:52.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:37:52.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b44c0 con 0x7f06f80a98f0 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f06e8001690 con 0x7f06f80a98f0 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06f6ffd640 10 monclient: dump: 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.289 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:52.290 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:52.290 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:52.290 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:52.290 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:52.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06fe75d640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:37:52.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.289+0000 7f06fe75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:52.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06e80017f0 con 0x7f06f80a98f0 2024-03-25T19:37:52.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:52.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06f6ffd640 1 -- 172.21.15.47:0/2607483783 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f06e8001c40 con 0x7f06f80a98f0 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06f6ffd640 10 monclient: dump: 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06fe75d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.306+0000 7f06fe75d640 4 set_mon_vals no callback set 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 10 monclient: shutdown 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 1 -- 172.21.15.47:0/2607483783 >> v1:172.21.15.142:6789/0 conn(0x7f06f80a98f0 legacy=0x7f06f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 1 -- 172.21.15.47:0/2607483783 shutdown_connections 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 1 -- 172.21.15.47:0/2607483783 >> 172.21.15.47:0/2607483783 conn(0x7f06f80b7ea0 msgr2=0x7f06f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 1 -- 172.21.15.47:0/2607483783 shutdown_connections 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.307+0000 7f06fe75d640 1 -- 172.21.15.47:0/2607483783 wait complete. 2024-03-25T19:37:52.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.308+0000 7f06fe75d640 10 monclient: build_initial_monmap 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.308+0000 7f06fe75d640 10 monclient: monmap: 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.308+0000 7f06fe75d640 1 librados: starting msgr at 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.308+0000 7f06fe75d640 1 librados: starting objecter 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.308+0000 7f06fe75d640 1 Processor -- start 2024-03-25T19:37:52.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.309+0000 7f06fe75d640 1 -- start start 2024-03-25T19:37:52.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.309+0000 7f06fe75d640 1 librados: setting wanted keys 2024-03-25T19:37:52.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.309+0000 7f06fe75d640 1 librados: calling monclient init 2024-03-25T19:37:52.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.309+0000 7f06fe75d640 10 monclient: init 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.309+0000 7f06fe75d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 10 monclient(hunting): picked mon.noname-a con 0x7f06f80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 10 monclient(hunting): picked mon.noname-b con 0x7f06f813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 10 monclient(hunting): picked mon.noname-c con 0x7f06f81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f8138b20 con 0x7f06f80a98f0 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f8137a80 con 0x7f06f813f710 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f8139880 con 0x7f06f81430c0 2024-03-25T19:37:52.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:52.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.310+0000 7f06fe75d640 10 monclient(hunting): authenticate will time out at 5787.596181s 2024-03-25T19:37:52.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.315+0000 7f06f77fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f06f813f710 0x7f06f8139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:52014/0 (socket says 172.21.15.47:52014) 2024-03-25T19:37:52.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.315+0000 7f06f77fe640 1 -- 172.21.15.47:0/2011271914 learned_addr learned my addr 172.21.15.47:0/2011271914 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.316+0000 7f06f4ff9640 10 client.?.objecter ms_handle_connect 0x7f06f813f710 2024-03-25T19:37:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.316+0000 7f06f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:52.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.329+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015888450 0 0) 0x7f06f8137a80 con 0x7f06f813f710 2024-03-25T19:37:52.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.329+0000 7f06f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.329+0000 7f06f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.329+0000 7f06f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.329+0000 7f06f4ff9640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:37:52.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.329+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e4004eb0 con 0x7f06f813f710 2024-03-25T19:37:52.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.336+0000 7f06f4ff9640 10 client.?.objecter ms_handle_connect 0x7f06f81430c0 2024-03-25T19:37:52.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.336+0000 7f06f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:52.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.337+0000 7f06f4ff9640 10 client.?.objecter ms_handle_connect 0x7f06f80a98f0 2024-03-25T19:37:52.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.337+0000 7f06f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:52.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.345+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206593555 0 0) 0x7f06e4004eb0 con 0x7f06f813f710 2024-03-25T19:37:52.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.346+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f8137a80 con 0x7f06f813f710 2024-03-25T19:37:52.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.346+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f06ec001a30 con 0x7f06f813f710 2024-03-25T19:37:52.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.346+0000 7f06f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.346+0000 7f06f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.346+0000 7f06f4ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:52.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.346+0000 7f06f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:52.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2600645919 0 0) 0x7f06f8137a80 con 0x7f06f813f710 2024-03-25T19:37:52.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 >> v1:172.21.15.47:6790/0 conn(0x7f06f81430c0 legacy=0x7f06f81e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 >> v1:172.21.15.47:6789/0 conn(0x7f06f80a98f0 legacy=0x7f06f8137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 1 monclient: found mon.b 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.366+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8139a10 con 0x7f06f813f710 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06f4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06f4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 5 monclient: authenticate success, global_id 4369 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 10 monclient: _renew_subs 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f81dab00 con 0x7f06f813f710 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 10 monclient: _renew_subs 2024-03-25T19:37:52.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:52.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f813b4f0 con 0x7f06f813f710 2024-03-25T19:37:52.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.367+0000 7f06fe75d640 1 librados: init done 2024-03-25T19:37:52.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.368+0000 7f06fe75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.368+0000 7f06fe75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.368+0000 7f06fe75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:52.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.368+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f80627b0 con 0x7f06f813f710 2024-03-25T19:37:52.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06ec001cb0 con 0x7f06f813f710 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f06ec003a60 con 0x7f06f813f710 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f4ff9640 10 monclient: dump: 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:52.645 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:52.646 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:52.646 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:52.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.644+0000 7f06f67fc640 4 set_mon_vals no callback set 2024-03-25T19:37:52.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.645+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f06ec003c40 con 0x7f06f813f710 2024-03-25T19:37:52.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.646+0000 7f06f7fff640 10 monclient: get_auth_request con 0x7f06e406dd90 auth_method 0 2024-03-25T19:37:52.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.646+0000 7f06f4ff9640 10 client.4369.objecter ms_handle_connect 0x7f06e406dd90 2024-03-25T19:37:52.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.648+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f06ec054860 con 0x7f06f813f710 2024-03-25T19:37:52.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.648+0000 7f06f4ff9640 10 client.4369.objecter ms_dispatch 0x7f06f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:52.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.648+0000 7f06f4ff9640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:52.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.648+0000 7f06f4ff9640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:52.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.648+0000 7f06f4ff9640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-25T19:37:52.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.650+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06ec050370 con 0x7f06f813f710 2024-03-25T19:37:52.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.650+0000 7f06f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.650+0000 7f06f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.922+0000 7f06fe75d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.922+0000 7f06fe75d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.922+0000 7f06fe75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:37:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.922+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f80b3e80 con 0x7f06f813f710 2024-03-25T19:37:52.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.951+0000 7f06f4ff9640 1 -- 172.21.15.47:0/2011271914 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f06ec0522a0 con 0x7f06f813f710 2024-03-25T19:37:52.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.951+0000 7f06f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.951+0000 7f06f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:52.951 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:37:52.951 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 10 librados: watch_flush enter 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 10 librados: watch_flush exit 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 10 client.4369.objecter successfully canceled tick 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 >> v1:172.21.15.142:6800/1025302268 conn(0x7f06e406dd90 legacy=0x7f06e4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 10 monclient: shutdown 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 >> v1:172.21.15.142:6789/0 conn(0x7f06f813f710 legacy=0x7f06f8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 shutdown_connections 2024-03-25T19:37:52.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 >> 172.21.15.47:0/2011271914 conn(0x7f06f80b7ea0 msgr2=0x7f06f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:52.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 shutdown_connections 2024-03-25T19:37:52.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 -- 172.21.15.47:0/2011271914 wait complete. 2024-03-25T19:37:52.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:52.954+0000 7f06fe75d640 1 librados: shutdown 2024-03-25T19:37:52.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:57.611 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:34.847 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.624 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:34.860 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.654 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:34.889 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.666 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:34.856 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.672 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:34.907 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.673 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:34.864 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.681 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:34.917 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.707 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:34.898 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:57.714 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:34.950 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.771 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:59.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.886+0000 7f38742ed640 10 monclient: get_monmap_and_config 2024-03-25T19:37:59.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.886+0000 7f38742ed640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.886+0000 7f38742ed640 10 monclient: monmap: 2024-03-25T19:37:59.885 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:37:59.886925+0000 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:37:59.886925+0000 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:37:59.886 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:59.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.886+0000 7f38742ed640 1 Processor -- start 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.886+0000 7f38742ed640 1 -- start start 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.886+0000 7f38742ed640 10 monclient: init 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.887+0000 7f38742ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.887+0000 7f38742ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.887+0000 7f38742ed640 10 monclient(hunting): picked mon.noname-b con 0x7f386c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.887+0000 7f38742ed640 10 monclient(hunting): picked mon.noname-c con 0x7f386c13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.888+0000 7f38742ed640 10 monclient(hunting): picked mon.noname-a con 0x7f386c1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.888+0000 7f38742ed640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c142520 con 0x7f386c1430c0 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.888+0000 7f38742ed640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c0b28a0 con 0x7f386c13f710 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.888+0000 7f38742ed640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c0b2a30 con 0x7f386c0a98f0 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.888+0000 7f38742ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.888+0000 7f38742ed640 10 monclient(hunting): authenticate will time out at 5795.174086s 2024-03-25T19:37:59.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.891+0000 7f3871888640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f386c13f710 0x7f386c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:51132/0 (socket says 172.21.15.47:51132) 2024-03-25T19:37:59.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.891+0000 7f3871888640 1 -- 172.21.15.47:0/3164942865 learned_addr learned my addr 172.21.15.47:0/3164942865 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:59.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.898+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463425769 0 0) 0x7f386c0b28a0 con 0x7f386c13f710 2024-03-25T19:37:59.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.898+0000 7f3871087640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.898+0000 7f3871087640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.898+0000 7f3871087640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.898+0000 7f3871087640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:37:59.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.898+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3858003520 con 0x7f386c13f710 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417500483 0 0) 0x7f3858003520 con 0x7f386c13f710 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 >> v1:172.21.15.47:6790/0 conn(0x7f386c0a98f0 legacy=0x7f386c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 >> v1:172.21.15.47:6789/0 conn(0x7f386c1430c0 legacy=0x7f386c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 1 monclient: found mon.noname-c 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c0b44c0 con 0x7f386c13f710 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f385c001690 con 0x7f386c13f710 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f3871087640 10 monclient: dump: 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f38742ed640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:37:59.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.908+0000 7f38742ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:59.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.925+0000 7f3871087640 1 -- 172.21.15.47:0/3164942865 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f385c001860 con 0x7f386c13f710 2024-03-25T19:37:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.925+0000 7f3871087640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.925+0000 7f38742ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.925+0000 7f38742ed640 4 set_mon_vals no callback set 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 10 monclient: shutdown 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 1 -- 172.21.15.47:0/3164942865 >> v1:172.21.15.142:6789/0 conn(0x7f386c13f710 legacy=0x7f386c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 1 -- 172.21.15.47:0/3164942865 shutdown_connections 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 1 -- 172.21.15.47:0/3164942865 >> 172.21.15.47:0/3164942865 conn(0x7f386c0b7ea0 msgr2=0x7f386c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 1 -- 172.21.15.47:0/3164942865 shutdown_connections 2024-03-25T19:37:59.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.926+0000 7f38742ed640 1 -- 172.21.15.47:0/3164942865 wait complete. 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.927+0000 7f38742ed640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.927+0000 7f38742ed640 10 monclient: monmap: 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:37:59.927 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.927+0000 7f38742ed640 1 librados: starting msgr at 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.927+0000 7f38742ed640 1 librados: starting objecter 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 Processor -- start 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 -- start start 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 librados: setting wanted keys 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 librados: calling monclient init 2024-03-25T19:37:59.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient: init 2024-03-25T19:37:59.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:59.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient(hunting): picked mon.noname-b con 0x7f386c0a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:37:59.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient(hunting): picked mon.noname-a con 0x7f386c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:37:59.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient(hunting): picked mon.noname-c con 0x7f386c1430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:37:59.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c0b0b30 con 0x7f386c13f710 2024-03-25T19:37:59.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c0aee30 con 0x7f386c0a98f0 2024-03-25T19:37:59.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c1db5f0 con 0x7f386c1430c0 2024-03-25T19:37:59.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.928+0000 7f38742ed640 10 monclient(hunting): authenticate will time out at 5795.214881s 2024-03-25T19:37:59.932 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.932+0000 7f387288a640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f386c1430c0 0x7f386c0af370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49524/0 (socket says 172.21.15.47:49524) 2024-03-25T19:37:59.932 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.932+0000 7f387288a640 1 -- 172.21.15.47:0/1987577745 learned_addr learned my addr 172.21.15.47:0/1987577745 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:37:59.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.934+0000 7f3862ffd640 10 client.?.objecter ms_handle_connect 0x7f386c1430c0 2024-03-25T19:37:59.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.934+0000 7f3862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.940+0000 7f3862ffd640 10 client.?.objecter ms_handle_connect 0x7f386c13f710 2024-03-25T19:37:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.940+0000 7f3862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138255244 0 0) 0x7f386c0b0b30 con 0x7f386c13f710 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3850004eb0 con 0x7f386c13f710 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 10 client.?.objecter ms_handle_connect 0x7f386c0a98f0 2024-03-25T19:37:59.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.950+0000 7f3862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.957+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568390119 0 0) 0x7f386c0aee30 con 0x7f386c0a98f0 2024-03-25T19:37:59.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.957+0000 7f3862ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.957+0000 7f3862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.957+0000 7f3862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.957+0000 7f3862ffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:37:59.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.957+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f386c0b0b30 con 0x7f386c0a98f0 2024-03-25T19:37:59.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.966+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148698164 0 0) 0x7f386c1db5f0 con 0x7f386c1430c0 2024-03-25T19:37:59.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.966+0000 7f3862ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.966+0000 7f3862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.966+0000 7f3862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.966+0000 7f3862ffd640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:37:59.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.966+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f386c0aee30 con 0x7f386c1430c0 2024-03-25T19:37:59.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185779935 0 0) 0x7f3850004eb0 con 0x7f386c13f710 2024-03-25T19:37:59.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f386c1db5f0 con 0x7f386c13f710 2024-03-25T19:37:59.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f385c001d50 con 0x7f386c13f710 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:59.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.969+0000 7f3862ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:59.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872828612 0 0) 0x7f386c0b0b30 con 0x7f386c0a98f0 2024-03-25T19:37:59.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3850004eb0 con 0x7f386c0a98f0 2024-03-25T19:37:59.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f3868001510 con 0x7f386c0a98f0 2024-03-25T19:37:59.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:59.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:37:59.975 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:37:59.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 156535706 0 0) 0x7f386c1db5f0 con 0x7f386c13f710 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 >> v1:172.21.15.47:6790/0 conn(0x7f386c1430c0 legacy=0x7f386c0af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 >> v1:172.21.15.142:6789/0 conn(0x7f386c0a98f0 legacy=0x7f386c1dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 monclient: found mon.a 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c1db780 con 0x7f386c13f710 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f3862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 10 monclient: _renew_subs 2024-03-25T19:37:59.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c0afa80 con 0x7f386c13f710 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 10 monclient: _renew_subs 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c14e380 con 0x7f386c13f710 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.975+0000 7f38742ed640 1 librados: init done 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.976+0000 7f38742ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.976+0000 7f38742ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.976+0000 7f38742ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:37:59.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:37:59.976+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c0afd80 con 0x7f386c13f710 2024-03-25T19:38:00.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f385c004a90 con 0x7f386c13f710 2024-03-25T19:38:00.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3862ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:00.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f385c001250 con 0x7f386c13f710 2024-03-25T19:38:00.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:00.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3862ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3862ffd640 10 monclient: dump: 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:00.308 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:00.309 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:00.309 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:00.309 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:00.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.307+0000 7f3870886640 4 set_mon_vals no callback set 2024-03-25T19:38:00.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.308+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f385c01ab30 con 0x7f386c13f710 2024-03-25T19:38:00.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.308+0000 7f3872089640 10 monclient: get_auth_request con 0x7f385006df30 auth_method 0 2024-03-25T19:38:00.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.309+0000 7f3862ffd640 10 client.4380.objecter ms_handle_connect 0x7f385006df30 2024-03-25T19:38:00.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f385c054750 con 0x7f386c13f710 2024-03-25T19:38:00.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 10 client.4380.objecter ms_dispatch 0x7f386c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:00.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:00.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:00.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-25T19:38:00.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f385c050090 con 0x7f386c13f710 2024-03-25T19:38:00.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:00.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.316+0000 7f3862ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.572+0000 7f38742ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.573+0000 7f38742ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.573+0000 7f38742ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:38:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.573+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f386c0627b0 con 0x7f386c13f710 2024-03-25T19:38:00.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.602+0000 7f3862ffd640 1 -- 172.21.15.47:0/1987577745 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f385c052190 con 0x7f386c13f710 2024-03-25T19:38:00.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.602+0000 7f3862ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.602 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.602+0000 7f3862ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:00.602 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:00.602 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:00.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.604+0000 7f38742ed640 10 librados: watch_flush enter 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.604+0000 7f38742ed640 10 librados: watch_flush exit 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.604+0000 7f38742ed640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.604+0000 7f38742ed640 10 client.4380.objecter successfully canceled tick 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 >> v1:172.21.15.142:6800/1025302268 conn(0x7f385006df30 legacy=0x7f38500703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 10 monclient: shutdown 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 >> v1:172.21.15.47:6789/0 conn(0x7f386c13f710 legacy=0x7f386c0ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 shutdown_connections 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 >> 172.21.15.47:0/1987577745 conn(0x7f386c0b7ea0 msgr2=0x7f386c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 shutdown_connections 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 -- 172.21.15.47:0/1987577745 wait complete. 2024-03-25T19:38:00.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:00.605+0000 7f38742ed640 1 librados: shutdown 2024-03-25T19:38:00.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:07.422 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:07.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.535+0000 7f1d63943640 10 monclient: get_monmap_and_config 2024-03-25T19:38:07.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.535+0000 7f1d63943640 10 monclient: build_initial_monmap 2024-03-25T19:38:07.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.536+0000 7f1d63943640 10 monclient: monmap: 2024-03-25T19:38:07.535 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:07.536761+0000 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:07.536761+0000 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:07.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.536+0000 7f1d63943640 1 Processor -- start 2024-03-25T19:38:07.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.536+0000 7f1d63943640 1 -- start start 2024-03-25T19:38:07.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.536+0000 7f1d63943640 10 monclient: init 2024-03-25T19:38:07.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:07.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:07.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c142520 con 0x7f1d5c1430c0 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0b28a0 con 0x7f1d5c13f710 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0b2a30 con 0x7f1d5c0a98f0 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:07.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.537+0000 7f1d63943640 10 monclient(hunting): authenticate will time out at 5802.823684s 2024-03-25T19:38:07.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.540+0000 7f1d61ee0640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1d5c1430c0 0x7f1d5c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36192/0 (socket says 172.21.15.47:36192) 2024-03-25T19:38:07.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.540+0000 7f1d61ee0640 1 -- 172.21.15.47:0/3413617336 learned_addr learned my addr 172.21.15.47:0/3413617336 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:07.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.555+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803629917 0 0) 0x7f1d5c142520 con 0x7f1d5c1430c0 2024-03-25T19:38:07.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.555+0000 7f1d53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.555+0000 7f1d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.555+0000 7f1d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.555+0000 7f1d53fff640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:38:07.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.555+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d48003520 con 0x7f1d5c1430c0 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.561+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66898905 0 0) 0x7f1d5c0b28a0 con 0x7f1d5c13f710 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.561+0000 7f1d53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.561+0000 7f1d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.561+0000 7f1d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.561+0000 7f1d53fff640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.561+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c142520 con 0x7f1d5c13f710 2024-03-25T19:38:07.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.566+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523650666 0 0) 0x7f1d5c0b2a30 con 0x7f1d5c0a98f0 2024-03-25T19:38:07.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.566+0000 7f1d53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.566+0000 7f1d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.566+0000 7f1d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.566+0000 7f1d53fff640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:38:07.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.566+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c0b28a0 con 0x7f1d5c0a98f0 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869611223 0 0) 0x7f1d5c142520 con 0x7f1d5c13f710 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 >> v1:172.21.15.47:6790/0 conn(0x7f1d5c0a98f0 legacy=0x7f1d5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 >> v1:172.21.15.47:6789/0 conn(0x7f1d5c1430c0 legacy=0x7f1d5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 1 monclient: found mon.noname-c 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0b44c0 con 0x7f1d5c13f710 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1d4c001670 con 0x7f1d5c13f710 2024-03-25T19:38:07.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d53fff640 10 monclient: dump: 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d63943640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:38:07.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.571+0000 7f1d63943640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.574+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d4c0017d0 con 0x7f1d5c13f710 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d53fff640 1 -- 172.21.15.47:0/3413617336 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1d4c001be0 con 0x7f1d5c13f710 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d53fff640 10 monclient: dump: 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:07.574 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d63943640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:07.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d63943640 4 set_mon_vals no callback set 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d63943640 10 monclient: shutdown 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d63943640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.575+0000 7f1d63943640 1 -- 172.21.15.47:0/3413617336 >> v1:172.21.15.142:6789/0 conn(0x7f1d5c13f710 legacy=0x7f1d5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.576+0000 7f1d63943640 1 -- 172.21.15.47:0/3413617336 shutdown_connections 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.576+0000 7f1d63943640 1 -- 172.21.15.47:0/3413617336 >> 172.21.15.47:0/3413617336 conn(0x7f1d5c0b7ea0 msgr2=0x7f1d5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.576+0000 7f1d63943640 1 -- 172.21.15.47:0/3413617336 shutdown_connections 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.576+0000 7f1d63943640 1 -- 172.21.15.47:0/3413617336 wait complete. 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 10 monclient: build_initial_monmap 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 10 monclient: monmap: 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:07.577 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 1 librados: starting msgr at 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 1 librados: starting objecter 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 1 Processor -- start 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 1 -- start start 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 1 librados: setting wanted keys 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 1 librados: calling monclient init 2024-03-25T19:38:07.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.577+0000 7f1d63943640 10 monclient: init 2024-03-25T19:38:07.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:07.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:07.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:07.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:07.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0abde0 con 0x7f1d5c13f710 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0ac0c0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c139250 con 0x7f1d5c0a98f0 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.578+0000 7f1d63943640 10 monclient(hunting): authenticate will time out at 5802.864691s 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.581+0000 7f1d61ee0640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1d5c1430c0 0x7f1d5c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:43566/0 (socket says 172.21.15.47:43566) 2024-03-25T19:38:07.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.581+0000 7f1d61ee0640 1 -- 172.21.15.47:0/4239107858 learned_addr learned my addr 172.21.15.47:0/4239107858 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:07.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.582+0000 7f1d51ffb640 10 client.?.objecter ms_handle_connect 0x7f1d5c1430c0 2024-03-25T19:38:07.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.582+0000 7f1d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.588+0000 7f1d51ffb640 10 client.?.objecter ms_handle_connect 0x7f1d5c13f710 2024-03-25T19:38:07.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.588+0000 7f1d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.591+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119855534 0 0) 0x7f1d5c0ac0c0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.591+0000 7f1d51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.591+0000 7f1d51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.591+0000 7f1d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.591+0000 7f1d51ffb640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:38:07.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.591+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40004eb0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.599+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315073898 0 0) 0x7f1d5c0abde0 con 0x7f1d5c13f710 2024-03-25T19:38:07.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.599+0000 7f1d51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.599+0000 7f1d51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.599+0000 7f1d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.599+0000 7f1d51ffb640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:38:07.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.599+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c0ac0c0 con 0x7f1d5c13f710 2024-03-25T19:38:07.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3644816232 0 0) 0x7f1d40004eb0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d5c0abde0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1d54001b30 con 0x7f1d5c1430c0 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:07.606 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:07.607 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:07.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.606+0000 7f1d51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:07.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.608+0000 7f1d51ffb640 10 client.?.objecter ms_handle_connect 0x7f1d5c0a98f0 2024-03-25T19:38:07.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.608+0000 7f1d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1748270539 0 0) 0x7f1d5c0abde0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 >> v1:172.21.15.47:6790/0 conn(0x7f1d5c0a98f0 legacy=0x7f1d5c138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 >> v1:172.21.15.47:6789/0 conn(0x7f1d5c13f710 legacy=0x7f1d5c0ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 1 monclient: found mon.b 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0acbf0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:07.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 10 monclient: _renew_subs 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c137de0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 10 monclient: _renew_subs 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c1368c0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.612+0000 7f1d63943640 1 librados: init done 2024-03-25T19:38:07.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.613+0000 7f1d63943640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.613+0000 7f1d63943640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.613+0000 7f1d63943640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:07.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.613+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5c136bc0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d54001fe0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1d54004b60 con 0x7f1d5c1430c0 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d51ffb640 10 monclient: dump: 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:07.890 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:07.891 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:07.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.889+0000 7f1d537fe640 4 set_mon_vals no callback set 2024-03-25T19:38:07.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.890+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f1d54004d80 con 0x7f1d5c1430c0 2024-03-25T19:38:07.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.891+0000 7f1d616df640 10 monclient: get_auth_request con 0x7f1d4006dda0 auth_method 0 2024-03-25T19:38:07.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.891+0000 7f1d51ffb640 10 client.4381.objecter ms_handle_connect 0x7f1d4006dda0 2024-03-25T19:38:07.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.894+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f1d540548d0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.894+0000 7f1d51ffb640 10 client.4381.objecter ms_dispatch 0x7f1d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:07.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.894+0000 7f1d51ffb640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:07.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.894+0000 7f1d51ffb640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:07.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.894+0000 7f1d51ffb640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T19:38:07.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.895+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d540503e0 con 0x7f1d5c1430c0 2024-03-25T19:38:07.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.895+0000 7f1d51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:07.895+0000 7f1d51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:08.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.165+0000 7f1d63943640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:08.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.165+0000 7f1d63943640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:08.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.165+0000 7f1d63943640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:08.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.165+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d5c14f2d0 con 0x7f1d5c1430c0 2024-03-25T19:38:08.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.192+0000 7f1d51ffb640 1 -- 172.21.15.47:0/4239107858 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f1d54052310 con 0x7f1d5c1430c0 2024-03-25T19:38:08.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.192+0000 7f1d51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:08.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.192+0000 7f1d51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:08.193 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:08.193 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:08.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.194+0000 7f1d63943640 10 librados: watch_flush enter 2024-03-25T19:38:08.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 10 librados: watch_flush exit 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 10 client.4381.objecter successfully canceled tick 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 >> v1:172.21.15.142:6800/1025302268 conn(0x7f1d4006dda0 legacy=0x7f1d40070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 10 monclient: shutdown 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 >> v1:172.21.15.142:6789/0 conn(0x7f1d5c1430c0 legacy=0x7f1d5c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 shutdown_connections 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 >> 172.21.15.47:0/4239107858 conn(0x7f1d5c0b7ea0 msgr2=0x7f1d5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 shutdown_connections 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 -- 172.21.15.47:0/4239107858 wait complete. 2024-03-25T19:38:08.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:08.195+0000 7f1d63943640 1 librados: shutdown 2024-03-25T19:38:08.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:15.014 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:15.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.131+0000 7fd3121a3640 10 monclient: get_monmap_and_config 2024-03-25T19:38:15.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.131+0000 7fd3121a3640 10 monclient: build_initial_monmap 2024-03-25T19:38:15.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.131+0000 7fd3121a3640 10 monclient: monmap: 2024-03-25T19:38:15.131 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:15.131 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:15.131 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:15.132916+0000 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:15.132916+0000 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:15.132 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 1 Processor -- start 2024-03-25T19:38:15.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 1 -- start start 2024-03-25T19:38:15.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 10 monclient: init 2024-03-25T19:38:15.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:15.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 10 monclient(hunting): picked mon.noname-b con 0x7fd30c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 10 monclient(hunting): picked mon.noname-a con 0x7fd30c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 10 monclient(hunting): picked mon.noname-c con 0x7fd30c1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c142520 con 0x7fd30c13f710 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.132+0000 7fd3121a3640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c0b28a0 con 0x7fd30c1430c0 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.133+0000 7fd3121a3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c0b2a30 con 0x7fd30c0a98f0 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.133+0000 7fd3121a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.133+0000 7fd3121a3640 10 monclient(hunting): authenticate will time out at 5810.419971s 2024-03-25T19:38:15.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.138+0000 7fd30b7fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd30c0a98f0 0x7fd30c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34328/0 (socket says 172.21.15.47:34328) 2024-03-25T19:38:15.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.138+0000 7fd30b7fe640 1 -- 172.21.15.47:0/3150566096 learned_addr learned my addr 172.21.15.47:0/3150566096 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.152+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499511757 0 0) 0x7fd30c0b2a30 con 0x7fd30c0a98f0 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.152+0000 7fd30a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.152+0000 7fd30a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.152+0000 7fd30a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.152+0000 7fd30a7fc640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.152+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f4003520 con 0x7fd30c0a98f0 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2370146722 0 0) 0x7fd2f4003520 con 0x7fd30c0a98f0 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 >> v1:172.21.15.142:6789/0 conn(0x7fd30c1430c0 legacy=0x7fd30c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 >> v1:172.21.15.47:6789/0 conn(0x7fd30c13f710 legacy=0x7fd30c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c0b44c0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd30a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd3121a3640 5 monclient: authenticate success, global_id 4370 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.162+0000 7fd3121a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.163+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd2fc001690 con 0x7fd30c0a98f0 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.163+0000 7fd30a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.163+0000 7fd30a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.163+0000 7fd30a7fc640 10 monclient: dump: 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.163+0000 7fd3121a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.168+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2fc001920 con 0x7fd30c0a98f0 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.168+0000 7fd30a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.168+0000 7fd30a7fc640 1 -- 172.21.15.47:0/3150566096 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd2fc001cb0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.168+0000 7fd30a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.168+0000 7fd30a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.168+0000 7fd30a7fc640 10 monclient: dump: 2024-03-25T19:38:15.169 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:15.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 4 set_mon_vals no callback set 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 10 monclient: shutdown 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 1 -- 172.21.15.47:0/3150566096 >> v1:172.21.15.47:6790/0 conn(0x7fd30c0a98f0 legacy=0x7fd30c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 1 -- 172.21.15.47:0/3150566096 shutdown_connections 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.169+0000 7fd3121a3640 1 -- 172.21.15.47:0/3150566096 >> 172.21.15.47:0/3150566096 conn(0x7fd30c0b7ea0 msgr2=0x7fd30c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.170+0000 7fd3121a3640 1 -- 172.21.15.47:0/3150566096 shutdown_connections 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.170+0000 7fd3121a3640 1 -- 172.21.15.47:0/3150566096 wait complete. 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.170+0000 7fd3121a3640 10 monclient: build_initial_monmap 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.170+0000 7fd3121a3640 10 monclient: monmap: 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:15.172 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 1 librados: starting msgr at 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 1 librados: starting objecter 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 1 Processor -- start 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 1 -- start start 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 1 librados: setting wanted keys 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 1 librados: calling monclient init 2024-03-25T19:38:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.171+0000 7fd3121a3640 10 monclient: init 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient(hunting): picked mon.noname-c con 0x7fd30c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient(hunting): picked mon.noname-a con 0x7fd30c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient(hunting): picked mon.noname-b con 0x7fd30c1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c14e910 con 0x7fd30c13f710 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c14e5c0 con 0x7fd30c1430c0 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c138b10 con 0x7fd30c0a98f0 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:15.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.172+0000 7fd3121a3640 10 monclient(hunting): authenticate will time out at 5810.459452s 2024-03-25T19:38:15.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.189+0000 7fd30b7fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd30c0a98f0 0x7fd30c138400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34384/0 (socket says 172.21.15.47:34384) 2024-03-25T19:38:15.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.189+0000 7fd30b7fe640 1 -- 172.21.15.47:0/1617419072 learned_addr learned my addr 172.21.15.47:0/1617419072 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:15.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.189+0000 7fd2f3fff640 10 client.?.objecter ms_handle_connect 0x7fd30c0a98f0 2024-03-25T19:38:15.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.189+0000 7fd2f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:15.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.204+0000 7fd2f3fff640 10 client.?.objecter ms_handle_connect 0x7fd30c1430c0 2024-03-25T19:38:15.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.204+0000 7fd2f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:15.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.205+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500402361 0 0) 0x7fd30c138b10 con 0x7fd30c0a98f0 2024-03-25T19:38:15.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.205+0000 7fd2f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.205+0000 7fd2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.205+0000 7fd2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.205+0000 7fd2f3fff640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:38:15.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.205+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f8004eb0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 10 client.?.objecter ms_handle_connect 0x7fd30c13f710 2024-03-25T19:38:15.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:15.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916026985 0 0) 0x7fd30c14e5c0 con 0x7fd30c1430c0 2024-03-25T19:38:15.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:38:15.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.217+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd30c138b10 con 0x7fd30c1430c0 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 388282021 0 0) 0x7fd2f8004eb0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd30c14e5c0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd2fc001650 con 0x7fd30c0a98f0 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 10 monclient(hunting): dump: 2024-03-25T19:38:15.225 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.225+0000 7fd2f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:15.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.237+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059096281 0 0) 0x7fd30c14e910 con 0x7fd30c13f710 2024-03-25T19:38:15.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.237+0000 7fd2f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.237+0000 7fd2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.237+0000 7fd2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.237+0000 7fd2f3fff640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:38:15.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.237+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f8004eb0 con 0x7fd30c13f710 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.238+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4097064831 0 0) 0x7fd30c138b10 con 0x7fd30c1430c0 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.238+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd30c14e910 con 0x7fd30c1430c0 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.238+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd304001b00 con 0x7fd30c1430c0 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.238+0000 7fd2f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.238+0000 7fd2f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.238+0000 7fd2f3fff640 10 monclient(hunting): dump: 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:15.239 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:15.240 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:15.240 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 700687174 0 0) 0x7fd30c14e5c0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 >> v1:172.21.15.142:6789/0 conn(0x7fd30c1430c0 legacy=0x7fd30c1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 >> v1:172.21.15.47:6789/0 conn(0x7fd30c13f710 legacy=0x7fd30c1d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 1 monclient: found mon.c 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c138db0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd2f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:15.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 5 monclient: authenticate success, global_id 4373 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 10 monclient: _renew_subs 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c139000 con 0x7fd30c0a98f0 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 10 monclient: _renew_subs 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c1e32a0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.240+0000 7fd3121a3640 1 librados: init done 2024-03-25T19:38:15.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.241+0000 7fd3121a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.241+0000 7fd3121a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.241+0000 7fd3121a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:15.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.241+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30c0b3ef0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2fc004bf0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd2fc004f00 con 0x7fd30c0a98f0 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 10 monclient: dump: 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.559 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd309ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.558+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fd2fc01aed0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.559+0000 7fd30affd640 10 monclient: get_auth_request con 0x7fd2f806df20 auth_method 0 2024-03-25T19:38:15.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.559+0000 7fd2f3fff640 10 client.4373.objecter ms_handle_connect 0x7fd2f806df20 2024-03-25T19:38:15.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.566+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fd2fc054ab0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.566+0000 7fd2f3fff640 10 client.4373.objecter ms_dispatch 0x7fd30c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:15.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.566+0000 7fd2f3fff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:15.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.566+0000 7fd2f3fff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:15.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.567+0000 7fd2f3fff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-25T19:38:15.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.567+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2fc0503f0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.567+0000 7fd2f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.567+0000 7fd2f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:15.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.791+0000 7fd3121a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.791+0000 7fd3121a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.791+0000 7fd3121a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:15.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.791+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd30c0627b0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.818+0000 7fd2f3fff640 1 -- 172.21.15.47:0/1617419072 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fd2fc0524f0 con 0x7fd30c0a98f0 2024-03-25T19:38:15.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.818+0000 7fd2f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.818+0000 7fd2f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:15.819 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:15.819 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 10 librados: watch_flush enter 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 10 librados: watch_flush exit 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 10 client.4373.objecter successfully canceled tick 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 >> v1:172.21.15.142:6800/1025302268 conn(0x7fd2f806df20 legacy=0x7fd2f80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 10 monclient: shutdown 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 >> v1:172.21.15.47:6790/0 conn(0x7fd30c0a98f0 legacy=0x7fd30c138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 shutdown_connections 2024-03-25T19:38:15.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 >> 172.21.15.47:0/1617419072 conn(0x7fd30c0b7ea0 msgr2=0x7fd30c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:15.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 shutdown_connections 2024-03-25T19:38:15.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 -- 172.21.15.47:0/1617419072 wait complete. 2024-03-25T19:38:15.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:15.821+0000 7fd3121a3640 1 librados: shutdown 2024-03-25T19:38:15.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:17.668 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:54.904 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.674 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:54.909 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.722 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:54.958 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.735 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:54.926 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.742 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:54.932 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.746 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:54.982 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.752 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:54.988 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.778 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:03:54.968 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:17.785 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:55.020 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:22.641 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:22.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.759+0000 7f2df2245640 10 monclient: get_monmap_and_config 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.759+0000 7f2df2245640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.759+0000 7f2df2245640 10 monclient: monmap: 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:22.760105+0000 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:22.760105+0000 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:22.759 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.760+0000 7f2df2245640 1 Processor -- start 2024-03-25T19:38:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.760+0000 7f2df2245640 1 -- start start 2024-03-25T19:38:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.760+0000 7f2df2245640 10 monclient: init 2024-03-25T19:38:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient(hunting): picked mon.noname-c con 0x7f2dec0a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient(hunting): picked mon.noname-b con 0x7f2dec13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient(hunting): picked mon.noname-a con 0x7f2dec1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec142520 con 0x7f2dec1430c0 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0b28a0 con 0x7f2dec0a98f0 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0b2a30 con 0x7f2dec13f710 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.761+0000 7f2df2245640 10 monclient(hunting): authenticate will time out at 5818.047288s 2024-03-25T19:38:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.776+0000 7f2debfff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2dec1430c0 0x7f2dec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45410/0 (socket says 172.21.15.47:45410) 2024-03-25T19:38:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.776+0000 7f2debfff640 1 -- 172.21.15.47:0/1210210209 learned_addr learned my addr 172.21.15.47:0/1210210209 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:22.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.790+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332824499 0 0) 0x7f2dec142520 con 0x7f2dec1430c0 2024-03-25T19:38:22.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.790+0000 7f2dea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.790+0000 7f2dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.790+0000 7f2dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.790+0000 7f2dea7fc640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:38:22.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.790+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd4003520 con 0x7f2dec1430c0 2024-03-25T19:38:22.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.796+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563818724 0 0) 0x7f2dec0b28a0 con 0x7f2dec0a98f0 2024-03-25T19:38:22.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.796+0000 7f2dea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.796+0000 7f2dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.796+0000 7f2dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.796+0000 7f2dea7fc640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:38:22.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.796+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dec142520 con 0x7f2dec0a98f0 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689517975 0 0) 0x7f2dd4003520 con 0x7f2dec1430c0 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 >> v1:172.21.15.47:6790/0 conn(0x7f2dec13f710 legacy=0x7f2dec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 >> v1:172.21.15.142:6789/0 conn(0x7f2dec0a98f0 legacy=0x7f2dec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 1 monclient: found mon.noname-a 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0b44c0 con 0x7f2dec1430c0 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2de4001a10 con 0x7f2dec1430c0 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2dea7fc640 10 monclient: dump: 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2df2245640 5 monclient: authenticate success, global_id 4395 2024-03-25T19:38:22.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.801+0000 7f2df2245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2de4001b70 con 0x7f2dec1430c0 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2dea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2dea7fc640 1 -- 172.21.15.47:0/1210210209 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2de4001f80 con 0x7f2dec1430c0 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2df2245640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2df2245640 4 set_mon_vals no callback set 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2dea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2dea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.812+0000 7f2dea7fc640 10 monclient: dump: 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:22.813 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:22.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 10 monclient: shutdown 2024-03-25T19:38:22.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 1 -- 172.21.15.47:0/1210210209 >> v1:172.21.15.47:6789/0 conn(0x7f2dec1430c0 legacy=0x7f2dec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 1 -- 172.21.15.47:0/1210210209 shutdown_connections 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 1 -- 172.21.15.47:0/1210210209 >> 172.21.15.47:0/1210210209 conn(0x7f2dec0b7ea0 msgr2=0x7f2dec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 1 -- 172.21.15.47:0/1210210209 shutdown_connections 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.813+0000 7f2df2245640 1 -- 172.21.15.47:0/1210210209 wait complete. 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.814+0000 7f2df2245640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.814+0000 7f2df2245640 10 monclient: monmap: 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:22.816 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.814+0000 7f2df2245640 1 librados: starting msgr at 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.814+0000 7f2df2245640 1 librados: starting objecter 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.814+0000 7f2df2245640 1 Processor -- start 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 1 -- start start 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 1 librados: setting wanted keys 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 1 librados: calling monclient init 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient: init 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient(hunting): picked mon.noname-a con 0x7f2dec0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient(hunting): picked mon.noname-c con 0x7f2dec13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient(hunting): picked mon.noname-b con 0x7f2dec1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:22.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0abb00 con 0x7f2dec0a98f0 2024-03-25T19:38:22.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0abde0 con 0x7f2dec1430c0 2024-03-25T19:38:22.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec138e20 con 0x7f2dec13f710 2024-03-25T19:38:22.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.815+0000 7f2df2245640 10 monclient(hunting): authenticate will time out at 5818.101710s 2024-03-25T19:38:22.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.822+0000 7f2debfff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2dec1430c0 0x7f2dec0ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:49272/0 (socket says 172.21.15.47:49272) 2024-03-25T19:38:22.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.822+0000 7f2debfff640 1 -- 172.21.15.47:0/1600788807 learned_addr learned my addr 172.21.15.47:0/1600788807 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:22.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.823+0000 7f2dd3fff640 10 client.?.objecter ms_handle_connect 0x7f2dec1430c0 2024-03-25T19:38:22.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.823+0000 7f2dd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.834 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784433731 0 0) 0x7f2dec0abde0 con 0x7f2dec1430c0 2024-03-25T19:38:22.834 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.834 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd8004eb0 con 0x7f2dec1430c0 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 10 client.?.objecter ms_handle_connect 0x7f2dec13f710 2024-03-25T19:38:22.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.835+0000 7f2dd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.840+0000 7f2dd3fff640 10 client.?.objecter ms_handle_connect 0x7f2dec0a98f0 2024-03-25T19:38:22.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.840+0000 7f2dd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.842+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785269971 0 0) 0x7f2dec138e20 con 0x7f2dec13f710 2024-03-25T19:38:22.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.842+0000 7f2dd3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.842+0000 7f2dd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.842+0000 7f2dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.842+0000 7f2dd3fff640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:38:22.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.842+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dec0abde0 con 0x7f2dec13f710 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070096894 0 0) 0x7f2dd8004eb0 con 0x7f2dec1430c0 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dec138e20 con 0x7f2dec1430c0 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2de4001ec0 con 0x7f2dec1430c0 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 10 monclient(hunting): dump: 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:22.852 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:22.853 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:22.853 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:22.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.852+0000 7f2dd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:22.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.857+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1334458302 0 0) 0x7f2dec0abde0 con 0x7f2dec13f710 2024-03-25T19:38:22.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.857+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd8004eb0 con 0x7f2dec13f710 2024-03-25T19:38:22.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.857+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2de00018b0 con 0x7f2dec13f710 2024-03-25T19:38:22.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.857+0000 7f2dd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.857+0000 7f2dd3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.857+0000 7f2dd3fff640 10 monclient(hunting): dump: 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:22.858 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:22.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.861+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444322014 0 0) 0x7f2dec0abb00 con 0x7f2dec0a98f0 2024-03-25T19:38:22.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.861+0000 7f2dd3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.861+0000 7f2dd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.861+0000 7f2dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.861+0000 7f2dd3fff640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:38:22.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.861+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dec0abde0 con 0x7f2dec0a98f0 2024-03-25T19:38:22.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2809757644 0 0) 0x7f2dec138e20 con 0x7f2dec1430c0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 >> v1:172.21.15.47:6790/0 conn(0x7f2dec13f710 legacy=0x7f2dec136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 >> v1:172.21.15.47:6789/0 conn(0x7f2dec0a98f0 legacy=0x7f2dec1385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 1 monclient: found mon.b 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec137b00 con 0x7f2dec1430c0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2dd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 10 monclient: _renew_subs 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec1e34b0 con 0x7f2dec1430c0 2024-03-25T19:38:22.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 10 monclient: _renew_subs 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec1e3820 con 0x7f2dec1430c0 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.873+0000 7f2df2245640 1 librados: init done 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.874+0000 7f2df2245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.874+0000 7f2df2245640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.874+0000 7f2df2245640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:22.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:22.874+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dec0627b0 con 0x7f2dec1430c0 2024-03-25T19:38:23.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2de4004e10 con 0x7f2dec1430c0 2024-03-25T19:38:23.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2dd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:23.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2de4005120 con 0x7f2dec1430c0 2024-03-25T19:38:23.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2dd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:23.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2dd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:23.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2dd3fff640 10 monclient: dump: 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:23.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.155+0000 7f2de9ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:23.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.157+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f2de401b0f0 con 0x7f2dec1430c0 2024-03-25T19:38:23.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.157+0000 7f2deb7fe640 10 monclient: get_auth_request con 0x7f2dd806df50 auth_method 0 2024-03-25T19:38:23.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.158+0000 7f2dd3fff640 10 client.4393.objecter ms_handle_connect 0x7f2dd806df50 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f2de4054b10 con 0x7f2dec1430c0 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 10 client.4393.objecter ms_dispatch 0x7f2dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2de4050620 con 0x7f2dec1430c0 2024-03-25T19:38:23.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:23.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.161+0000 7f2dd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:23.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.408+0000 7f2df2245640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:23.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.408+0000 7f2df2245640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:23.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.408+0000 7f2df2245640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:23.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.408+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dec0b1760 con 0x7f2dec1430c0 2024-03-25T19:38:23.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.458+0000 7f2dd3fff640 1 -- 172.21.15.47:0/1600788807 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f2de4052550 con 0x7f2dec1430c0 2024-03-25T19:38:23.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.458+0000 7f2dd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:23.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.458+0000 7f2dd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:23.457 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:23.457 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 10 librados: watch_flush enter 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 10 librados: watch_flush exit 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 10 client.4393.objecter successfully canceled tick 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 >> v1:172.21.15.142:6800/1025302268 conn(0x7f2dd806df50 legacy=0x7f2dd8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 10 monclient: shutdown 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 >> v1:172.21.15.142:6789/0 conn(0x7f2dec1430c0 legacy=0x7f2dec0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 shutdown_connections 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.460+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 >> 172.21.15.47:0/1600788807 conn(0x7f2dec0b7ea0 msgr2=0x7f2dec14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.461+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 shutdown_connections 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.461+0000 7f2df2245640 1 -- 172.21.15.47:0/1600788807 wait complete. 2024-03-25T19:38:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:23.461+0000 7f2df2245640 1 librados: shutdown 2024-03-25T19:38:23.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:30.278 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.378+0000 7f6f81df6640 10 monclient: get_monmap_and_config 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.378+0000 7f6f81df6640 10 monclient: build_initial_monmap 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.378+0000 7f6f81df6640 10 monclient: monmap: 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:30.378842+0000 2024-03-25T19:38:30.377 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:30.378842+0000 2024-03-25T19:38:30.378 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:30.378 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:30.378 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:30.378 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:30.378 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:30.378 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.378+0000 7f6f81df6640 1 Processor -- start 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 1 -- start start 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient: init 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7c0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:30.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7c13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7c1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c142520 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c0b28a0 con 0x7f6f7c1430c0 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c0b2a30 con 0x7f6f7c13f710 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.379+0000 7f6f81df6640 10 monclient(hunting): authenticate will time out at 5825.665645s 2024-03-25T19:38:30.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.388+0000 7f6f7b7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6f7c0a98f0 0x7f6f7c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40600/0 (socket says 172.21.15.47:40600) 2024-03-25T19:38:30.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.388+0000 7f6f7b7fe640 1 -- 172.21.15.47:0/1075455813 learned_addr learned my addr 172.21.15.47:0/1075455813 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:30.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.395+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31094056 0 0) 0x7f6f7c142520 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.396+0000 7f6f7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.396+0000 7f6f7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.396+0000 7f6f7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.396+0000 7f6f7affd640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:38:30.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.396+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f68003520 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 791998897 0 0) 0x7f6f68003520 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 >> v1:172.21.15.47:6790/0 conn(0x7f6f7c13f710 legacy=0x7f6f7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 >> v1:172.21.15.142:6789/0 conn(0x7f6f7c1430c0 legacy=0x7f6f7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 1 monclient: found mon.noname-a 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:38:30.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0b44c0 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: _finish_auth 0 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6f6c001690 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f81df6640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f81df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f7affd640 10 monclient: dump: 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.403 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.402+0000 7f6f81df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:30.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f6c001860 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:30.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f7affd640 1 -- 172.21.15.47:0/1075455813 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6f6c001cb0 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f7affd640 10 monclient: dump: 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f81df6640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:30.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.423+0000 7f6f81df6640 4 set_mon_vals no callback set 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.424+0000 7f6f81df6640 10 monclient: shutdown 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.424+0000 7f6f81df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.424+0000 7f6f81df6640 1 -- 172.21.15.47:0/1075455813 >> v1:172.21.15.47:6789/0 conn(0x7f6f7c0a98f0 legacy=0x7f6f7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.424+0000 7f6f81df6640 1 -- 172.21.15.47:0/1075455813 shutdown_connections 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.425+0000 7f6f81df6640 1 -- 172.21.15.47:0/1075455813 >> 172.21.15.47:0/1075455813 conn(0x7f6f7c0b7ea0 msgr2=0x7f6f7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.425+0000 7f6f81df6640 1 -- 172.21.15.47:0/1075455813 shutdown_connections 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.425+0000 7f6f81df6640 1 -- 172.21.15.47:0/1075455813 wait complete. 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.425+0000 7f6f81df6640 10 monclient: build_initial_monmap 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 10 monclient: monmap: 2024-03-25T19:38:30.426 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 1 librados: starting msgr at 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 1 librados: starting objecter 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 1 Processor -- start 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 1 -- start start 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 1 librados: setting wanted keys 2024-03-25T19:38:30.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 1 librados: calling monclient init 2024-03-25T19:38:30.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.426+0000 7f6f81df6640 10 monclient: init 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7c0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7c13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7c1430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c1d76e0 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c0abd80 con 0x7f6f7c13f710 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c137c90 con 0x7f6f7c1430c0 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:30.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.427+0000 7f6f81df6640 10 monclient(hunting): authenticate will time out at 5825.713632s 2024-03-25T19:38:30.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.442+0000 7f6f73fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f6f7c13f710 0x7f6f7c138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:55990/0 (socket says 172.21.15.47:55990) 2024-03-25T19:38:30.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.442+0000 7f6f73fff640 1 -- 172.21.15.47:0/1239475508 learned_addr learned my addr 172.21.15.47:0/1239475508 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:30.442 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.443+0000 7f6f78ff9640 10 client.?.objecter ms_handle_connect 0x7f6f7c13f710 2024-03-25T19:38:30.442 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.443+0000 7f6f78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024698308 0 0) 0x7f6f7c0abd80 con 0x7f6f7c13f710 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f7c13f710 2024-03-25T19:38:30.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 10 client.?.objecter ms_handle_connect 0x7f6f7c0a98f0 2024-03-25T19:38:30.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.453+0000 7f6f78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:30.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.458+0000 7f6f78ff9640 10 client.?.objecter ms_handle_connect 0x7f6f7c1430c0 2024-03-25T19:38:30.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.458+0000 7f6f78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:30.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.466+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529642287 0 0) 0x7f6f7c1d76e0 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.466+0000 7f6f78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.466+0000 7f6f78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.466+0000 7f6f78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.466+0000 7f6f78ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:38:30.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.466+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c0abd80 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.470+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490944631 0 0) 0x7f6f7c137c90 con 0x7f6f7c1430c0 2024-03-25T19:38:30.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.470+0000 7f6f78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.470+0000 7f6f78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.470+0000 7f6f78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.470+0000 7f6f78ff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:38:30.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.470+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c1d76e0 con 0x7f6f7c1430c0 2024-03-25T19:38:30.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.474+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2768330163 0 0) 0x7f6f5c004eb0 con 0x7f6f7c13f710 2024-03-25T19:38:30.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.474+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f7c137c90 con 0x7f6f7c13f710 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.474+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6f64001a20 con 0x7f6f7c13f710 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.474+0000 7f6f78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.474+0000 7f6f78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.474+0000 7f6f78ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.474 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.475 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:30.475 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:30.475 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:30.475 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:30.475 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.475+0000 7f6f78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.483+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335522259 0 0) 0x7f6f7c0abd80 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.483+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.483+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6f6c001b70 con 0x7f6f7c0a98f0 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.483+0000 7f6f78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.483+0000 7f6f78ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.483+0000 7f6f78ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.483 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:30.484 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:30.484 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:30.484 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:30.484 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3191835714 0 0) 0x7f6f7c137c90 con 0x7f6f7c13f710 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 >> v1:172.21.15.47:6790/0 conn(0x7f6f7c1430c0 legacy=0x7f6f7c1e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 >> v1:172.21.15.47:6789/0 conn(0x7f6f7c0a98f0 legacy=0x7f6f7c1d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 1 monclient: found mon.b 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:30.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c1d41e0 con 0x7f6f7c13f710 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 10 monclient: _renew_subs 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c1d82d0 con 0x7f6f7c13f710 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 10 monclient: _renew_subs 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c138d80 con 0x7f6f7c13f710 2024-03-25T19:38:30.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.486+0000 7f6f81df6640 1 librados: init done 2024-03-25T19:38:30.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.488+0000 7f6f81df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.488+0000 7f6f81df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.488+0000 7f6f81df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:30.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.488+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c0627b0 con 0x7f6f7c13f710 2024-03-25T19:38:30.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f64001b80 con 0x7f6f7c13f710 2024-03-25T19:38:30.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:30.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6f64003a40 con 0x7f6f7c13f710 2024-03-25T19:38:30.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:30.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 10 monclient: dump: 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:30.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f7a7fc640 4 set_mon_vals no callback set 2024-03-25T19:38:30.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.764+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f6f64003c60 con 0x7f6f7c13f710 2024-03-25T19:38:30.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.765+0000 7f6f7b7fe640 10 monclient: get_auth_request con 0x7f6f5c06df60 auth_method 0 2024-03-25T19:38:30.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.765+0000 7f6f78ff9640 10 client.4396.objecter ms_handle_connect 0x7f6f5c06df60 2024-03-25T19:38:30.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.776+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f6f640547f0 con 0x7f6f7c13f710 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.776+0000 7f6f78ff9640 10 client.4396.objecter ms_dispatch 0x7f6f7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.776+0000 7f6f78ff9640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.776+0000 7f6f78ff9640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.776+0000 7f6f78ff9640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.777+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f64050300 con 0x7f6f7c13f710 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.777+0000 7f6f78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:30.777+0000 7f6f78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:31.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.041+0000 7f6f81df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:31.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.041+0000 7f6f81df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:31.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.041+0000 7f6f81df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:31.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.041+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f7c13b430 con 0x7f6f7c13f710 2024-03-25T19:38:31.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.077+0000 7f6f78ff9640 1 -- 172.21.15.47:0/1239475508 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f6f64052230 con 0x7f6f7c13f710 2024-03-25T19:38:31.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.077+0000 7f6f78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:31.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.077+0000 7f6f78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:31.077 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:31.077 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:31.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 10 librados: watch_flush enter 2024-03-25T19:38:31.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 10 librados: watch_flush exit 2024-03-25T19:38:31.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:38:31.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 >> v1:172.21.15.142:6800/1025302268 conn(0x7f6f5c06df60 legacy=0x7f6f5c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 10 monclient: shutdown 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 >> v1:172.21.15.142:6789/0 conn(0x7f6f7c13f710 legacy=0x7f6f7c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 shutdown_connections 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 >> 172.21.15.47:0/1239475508 conn(0x7f6f7c0b7ea0 msgr2=0x7f6f7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.080+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 shutdown_connections 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.081+0000 7f6f81df6640 1 -- 172.21.15.47:0/1239475508 wait complete. 2024-03-25T19:38:31.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:31.081+0000 7f6f81df6640 1 librados: shutdown 2024-03-25T19:38:31.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:37.901 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.016+0000 7f5ba4693640 10 monclient: get_monmap_and_config 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.016+0000 7f5ba4693640 10 monclient: build_initial_monmap 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.016+0000 7f5ba4693640 10 monclient: monmap: 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:38.017654+0000 2024-03-25T19:38:38.016 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:38.017654+0000 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.018+0000 7f5ba4693640 1 Processor -- start 2024-03-25T19:38:38.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.018+0000 7f5ba4693640 1 -- start start 2024-03-25T19:38:38.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.018+0000 7f5ba4693640 10 monclient: init 2024-03-25T19:38:38.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:38.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:38.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient(hunting): picked mon.noname-c con 0x7f5b9c0a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient(hunting): picked mon.noname-b con 0x7f5b9c13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9c1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c142520 con 0x7f5b9c1430c0 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c0b28a0 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c0b2a30 con 0x7f5b9c13f710 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:38.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.019+0000 7f5ba4693640 10 monclient(hunting): authenticate will time out at 5833.305385s 2024-03-25T19:38:38.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.032+0000 7f5ba242f640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f5b9c0a98f0 0x7f5b9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:58796/0 (socket says 172.21.15.47:58796) 2024-03-25T19:38:38.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.032+0000 7f5ba242f640 1 -- 172.21.15.47:0/786647234 learned_addr learned my addr 172.21.15.47:0/786647234 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:38.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.043+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322102848 0 0) 0x7f5b9c0b28a0 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.043+0000 7f5ba142d640 10 monclient(hunting): _negotiate 2024-03-25T19:38:38.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.043+0000 7f5ba142d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:38.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.043+0000 7f5ba142d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:38.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.043+0000 7f5ba142d640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:38:38.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.043+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b88003520 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.049+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334774271 0 0) 0x7f5b9c0b2a30 con 0x7f5b9c13f710 2024-03-25T19:38:38.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.049+0000 7f5ba142d640 10 monclient(hunting): _negotiate 2024-03-25T19:38:38.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.049+0000 7f5ba142d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:38.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.049+0000 7f5ba142d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:38.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.049+0000 7f5ba142d640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:38:38.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.049+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b9c0b28a0 con 0x7f5b9c13f710 2024-03-25T19:38:38.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.064+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3006704687 0 0) 0x7f5b88003520 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.064+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 >> v1:172.21.15.47:6790/0 conn(0x7f5b9c13f710 legacy=0x7f5b9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 >> v1:172.21.15.47:6789/0 conn(0x7f5b9c1430c0 legacy=0x7f5b9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: _finish_hunting 0 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 1 monclient: found mon.noname-c 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c0b44c0 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: _finish_auth 0 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: _check_auth_tickets 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b8c001690 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba142d640 10 monclient: dump: 2024-03-25T19:38:38.065 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba4693640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:38:38.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.065+0000 7f5ba4693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:38.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.084+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b8c0017f0 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.084+0000 7f5ba142d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:38.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba4693640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:38.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.084+0000 7f5ba142d640 1 -- 172.21.15.47:0/786647234 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b8c001c00 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba142d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba4693640 4 set_mon_vals no callback set 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba142d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba142d640 10 monclient: dump: 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:38.085 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba4693640 10 monclient: shutdown 2024-03-25T19:38:38.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba4693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:38.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.085+0000 7f5ba4693640 1 -- 172.21.15.47:0/786647234 >> v1:172.21.15.142:6789/0 conn(0x7f5b9c0a98f0 legacy=0x7f5b9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.086+0000 7f5ba4693640 1 -- 172.21.15.47:0/786647234 shutdown_connections 2024-03-25T19:38:38.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.086+0000 7f5ba4693640 1 -- 172.21.15.47:0/786647234 >> 172.21.15.47:0/786647234 conn(0x7f5b9c0b7ea0 msgr2=0x7f5b9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:38.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.086+0000 7f5ba4693640 1 -- 172.21.15.47:0/786647234 shutdown_connections 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.086+0000 7f5ba4693640 1 -- 172.21.15.47:0/786647234 wait complete. 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.086+0000 7f5ba4693640 10 monclient: build_initial_monmap 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.086+0000 7f5ba4693640 10 monclient: monmap: 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 1 librados: starting msgr at 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 1 librados: starting objecter 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 1 Processor -- start 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 1 -- start start 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 1 librados: setting wanted keys 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 1 librados: calling monclient init 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.087+0000 7f5ba4693640 10 monclient: init 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient(hunting): picked mon.noname-c con 0x7f5b9c0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9c13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient(hunting): picked mon.noname-b con 0x7f5b9c1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:38.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c139a40 con 0x7f5b9c13f710 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c133f20 con 0x7f5b9c1430c0 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c0ab950 con 0x7f5b9c0a98f0 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.088+0000 7f5ba4693640 10 monclient(hunting): authenticate will time out at 5833.374678s 2024-03-25T19:38:38.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.098+0000 7f5ba2c30640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f5b9c1430c0 0x7f5b9c133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:58846/0 (socket says 172.21.15.47:58846) 2024-03-25T19:38:38.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.098+0000 7f5ba2c30640 1 -- 172.21.15.47:0/834819577 learned_addr learned my addr 172.21.15.47:0/834819577 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:38.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.101+0000 7f5b92ffd640 10 client.?.objecter ms_handle_connect 0x7f5b9c1430c0 2024-03-25T19:38:38.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.101+0000 7f5b92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:38.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.109+0000 7f5b92ffd640 10 client.?.objecter ms_handle_connect 0x7f5b9c13f710 2024-03-25T19:38:38.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.109+0000 7f5b92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:38.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.116+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924758585 0 0) 0x7f5b9c133f20 con 0x7f5b9c1430c0 2024-03-25T19:38:38.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.116+0000 7f5b92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:38.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.116+0000 7f5b92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:38.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.116+0000 7f5b92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:38.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.116+0000 7f5b92ffd640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:38:38.116 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.116+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b80004eb0 con 0x7f5b9c1430c0 2024-03-25T19:38:38.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.118+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624622567 0 0) 0x7f5b9c139a40 con 0x7f5b9c13f710 2024-03-25T19:38:38.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.118+0000 7f5b92ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:38.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.118+0000 7f5b92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:38.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.118+0000 7f5b92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:38.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.118+0000 7f5b92ffd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:38:38.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.118+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b9c133f20 con 0x7f5b9c13f710 2024-03-25T19:38:38.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.121+0000 7f5b92ffd640 10 client.?.objecter ms_handle_connect 0x7f5b9c0a98f0 2024-03-25T19:38:38.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.121+0000 7f5b92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.128+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951602465 0 0) 0x7f5b9c133f20 con 0x7f5b9c13f710 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.128+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b9c139a40 con 0x7f5b9c13f710 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.128+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b98001930 con 0x7f5b9c13f710 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.128+0000 7f5b92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.128+0000 7f5b92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.128+0000 7f5b92ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:38.128 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.129+0000 7f5b92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.130+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018758890 0 0) 0x7f5b80004eb0 con 0x7f5b9c1430c0 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.130+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b9c133f20 con 0x7f5b9c1430c0 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.130+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b94001c90 con 0x7f5b9c1430c0 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.130+0000 7f5b92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.130+0000 7f5b92ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.130+0000 7f5b92ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:38.130 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:38.131 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2762638259 0 0) 0x7f5b9c133f20 con 0x7f5b9c1430c0 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 >> v1:172.21.15.47:6790/0 conn(0x7f5b9c0a98f0 legacy=0x7f5b9c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 >> v1:172.21.15.47:6789/0 conn(0x7f5b9c13f710 legacy=0x7f5b9c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 1 monclient: found mon.b 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c0abc30 con 0x7f5b9c1430c0 2024-03-25T19:38:38.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 10 monclient: _renew_subs 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9c0abf10 con 0x7f5b9c1430c0 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 10 monclient: _renew_subs 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c139370 con 0x7f5b9c1430c0 2024-03-25T19:38:38.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.135+0000 7f5ba4693640 1 librados: init done 2024-03-25T19:38:38.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.136+0000 7f5ba4693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:38.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.136+0000 7f5ba4693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:38.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.136+0000 7f5ba4693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:38.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.136+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9c0627b0 con 0x7f5b9c1430c0 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b94003990 con 0x7f5b9c1430c0 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5b92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5b94003d60 con 0x7f5b9c1430c0 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5b92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5b92ffd640 10 monclient: dump: 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.413+0000 7f5ba0c2c640 4 set_mon_vals no callback set 2024-03-25T19:38:38.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.414+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f5b94003f80 con 0x7f5b9c1430c0 2024-03-25T19:38:38.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.414+0000 7f5ba242f640 10 monclient: get_auth_request con 0x7f5b8006de90 auth_method 0 2024-03-25T19:38:38.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.414+0000 7f5b92ffd640 10 client.4402.objecter ms_handle_connect 0x7f5b8006de90 2024-03-25T19:38:38.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f5b94054af0 con 0x7f5b9c1430c0 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 10 client.4402.objecter ms_dispatch 0x7f5b9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b94050600 con 0x7f5b9c1430c0 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:38.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.420+0000 7f5b92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:38.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.677+0000 7f5ba4693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.677+0000 7f5ba4693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.677+0000 7f5ba4693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:38:38.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.677+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b9c0b7180 con 0x7f5b9c1430c0 2024-03-25T19:38:38.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.707+0000 7f5b92ffd640 1 -- 172.21.15.47:0/834819577 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f5b94052530 con 0x7f5b9c1430c0 2024-03-25T19:38:38.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.707+0000 7f5b92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.707+0000 7f5b92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:38.707 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:38.707 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 10 librados: watch_flush enter 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 10 librados: watch_flush exit 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 >> v1:172.21.15.142:6800/1025302268 conn(0x7f5b8006de90 legacy=0x7f5b80070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 10 monclient: shutdown 2024-03-25T19:38:38.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:38.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 >> v1:172.21.15.142:6789/0 conn(0x7f5b9c1430c0 legacy=0x7f5b9c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 shutdown_connections 2024-03-25T19:38:38.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 >> 172.21.15.47:0/834819577 conn(0x7f5b9c0b7ea0 msgr2=0x7f5b9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:38.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 shutdown_connections 2024-03-25T19:38:38.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 -- 172.21.15.47:0/834819577 wait complete. 2024-03-25T19:38:38.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:38.710+0000 7f5ba4693640 1 librados: shutdown 2024-03-25T19:38:38.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:45.530 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:45.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.649+0000 7fa5d4238640 10 monclient: get_monmap_and_config 2024-03-25T19:38:45.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.649+0000 7fa5d4238640 10 monclient: build_initial_monmap 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.649+0000 7fa5d4238640 10 monclient: monmap: 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:45.650263+0000 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:45.650263+0000 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:45.649 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:45.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.650+0000 7fa5d4238640 1 Processor -- start 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.650+0000 7fa5d4238640 1 -- start start 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.650+0000 7fa5d4238640 10 monclient: init 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.650+0000 7fa5d4238640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.650+0000 7fa5d4238640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 10 monclient(hunting): picked mon.noname-c con 0x7fa5cc06ab80 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 10 monclient(hunting): picked mon.noname-a con 0x7fa5cc06af60 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 10 monclient(hunting): picked mon.noname-b con 0x7fa5cc0b00b0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc0b0500 con 0x7fa5cc06af60 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc13ffa0 con 0x7fa5cc06ab80 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc0af520 con 0x7fa5cc0b00b0 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.651+0000 7fa5d4238640 10 monclient(hunting): authenticate will time out at 5840.937430s 2024-03-25T19:38:45.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.656+0000 7fa5d17d3640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa5cc06af60 0x7fa5cc13f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56306/0 (socket says 172.21.15.47:56306) 2024-03-25T19:38:45.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.656+0000 7fa5d17d3640 1 -- 172.21.15.47:0/3139459109 learned_addr learned my addr 172.21.15.47:0/3139459109 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:45.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.666+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425974574 0 0) 0x7fa5cc0b0500 con 0x7fa5cc06af60 2024-03-25T19:38:45.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.666+0000 7fa5d0fd2640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.666+0000 7fa5d0fd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.666+0000 7fa5d0fd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.666+0000 7fa5d0fd2640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:38:45.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.667+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b8003520 con 0x7fa5cc06af60 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.672+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237728900 0 0) 0x7fa5cc13ffa0 con 0x7fa5cc06ab80 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.672+0000 7fa5d0fd2640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.672+0000 7fa5d0fd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.672+0000 7fa5d0fd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.672+0000 7fa5d0fd2640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.672+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5cc0b0500 con 0x7fa5cc06ab80 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3297354763 0 0) 0x7fa5cc0b0500 con 0x7fa5cc06ab80 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 >> v1:172.21.15.47:6790/0 conn(0x7fa5cc0b00b0 legacy=0x7fa5cc14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 >> v1:172.21.15.47:6789/0 conn(0x7fa5cc06af60 legacy=0x7fa5cc13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 1 monclient: found mon.noname-c 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc072890 con 0x7fa5cc06ab80 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa5c8001670 con 0x7fa5cc06ab80 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d0fd2640 10 monclient: dump: 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d4238640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.683+0000 7fa5d4238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5c80017d0 con 0x7fa5cc06ab80 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d0fd2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d0fd2640 1 -- 172.21.15.47:0/3139459109 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa5c8001c20 con 0x7fa5cc06ab80 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d4238640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d4238640 4 set_mon_vals no callback set 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d0fd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d0fd2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.688+0000 7fa5d0fd2640 10 monclient: dump: 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:45.689 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 10 monclient: shutdown 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 1 -- 172.21.15.47:0/3139459109 >> v1:172.21.15.142:6789/0 conn(0x7fa5cc06ab80 legacy=0x7fa5cc0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 1 -- 172.21.15.47:0/3139459109 shutdown_connections 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 1 -- 172.21.15.47:0/3139459109 >> 172.21.15.47:0/3139459109 conn(0x7fa5cc0b4a80 msgr2=0x7fa5cc0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 1 -- 172.21.15.47:0/3139459109 shutdown_connections 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.689+0000 7fa5d4238640 1 -- 172.21.15.47:0/3139459109 wait complete. 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 10 monclient: build_initial_monmap 2024-03-25T19:38:45.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 10 monclient: monmap: 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 1 librados: starting msgr at 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 1 librados: starting objecter 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 1 Processor -- start 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 1 -- start start 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 1 librados: setting wanted keys 2024-03-25T19:38:45.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 1 librados: calling monclient init 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.690+0000 7fa5d4238640 10 monclient: init 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.691+0000 7fa5d4238640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.691+0000 7fa5d4238640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.691+0000 7fa5d4238640 10 monclient(hunting): picked mon.noname-b con 0x7fa5cc06ab80 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.691+0000 7fa5d4238640 10 monclient(hunting): picked mon.noname-a con 0x7fa5cc0b00b0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.691+0000 7fa5d4238640 10 monclient(hunting): picked mon.noname-c con 0x7fa5cc1465a0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.691+0000 7fa5d4238640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc143f40 con 0x7fa5cc0b00b0 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.692+0000 7fa5d4238640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc1d7540 con 0x7fa5cc06ab80 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.692+0000 7fa5d4238640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc1d71d0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.692+0000 7fa5d4238640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:45.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.692+0000 7fa5d4238640 10 monclient(hunting): authenticate will time out at 5840.978051s 2024-03-25T19:38:45.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.697+0000 7fa5d27d5640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa5cc1465a0 0x7fa5cc1e1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:37546/0 (socket says 172.21.15.47:37546) 2024-03-25T19:38:45.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.697+0000 7fa5d27d5640 1 -- 172.21.15.47:0/1040223438 learned_addr learned my addr 172.21.15.47:0/1040223438 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:45.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.697+0000 7fa5c27fc640 10 client.?.objecter ms_handle_connect 0x7fa5cc1465a0 2024-03-25T19:38:45.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.697+0000 7fa5c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.703+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305162186 0 0) 0x7fa5cc1d71d0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.703+0000 7fa5c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.703+0000 7fa5c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.703+0000 7fa5c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.703+0000 7fa5c27fc640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.703+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b0004eb0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.704+0000 7fa5c27fc640 10 client.?.objecter ms_handle_connect 0x7fa5cc0b00b0 2024-03-25T19:38:45.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.704+0000 7fa5c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.706+0000 7fa5c27fc640 10 client.?.objecter ms_handle_connect 0x7fa5cc06ab80 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.706+0000 7fa5c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664619793 0 0) 0x7fa5b0004eb0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5cc1d71d0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa5c4001c90 con 0x7fa5cc1465a0 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 10 monclient(hunting): dump: 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:45.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.710+0000 7fa5c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3634354300 0 0) 0x7fa5cc1d71d0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 >> v1:172.21.15.142:6789/0 conn(0x7fa5cc06ab80 legacy=0x7fa5cc1d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 >> v1:172.21.15.47:6789/0 conn(0x7fa5cc0b00b0 legacy=0x7fa5cc143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 1 monclient: found mon.c 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc142f90 con 0x7fa5cc1465a0 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 10 monclient: _renew_subs 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc1431e0 con 0x7fa5cc1465a0 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 10 monclient: _renew_subs 2024-03-25T19:38:45.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc1e3340 con 0x7fa5cc1465a0 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.716+0000 7fa5d4238640 1 librados: init done 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.717+0000 7fa5d4238640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.717+0000 7fa5d4238640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.717+0000 7fa5d4238640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:45.717+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc0627b0 con 0x7fa5cc1465a0 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5c4003990 con 0x7fa5cc1465a0 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa5c4003da0 con 0x7fa5cc1465a0 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c27fc640 10 monclient: dump: 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:46.029 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:46.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.029+0000 7fa5c3fff640 4 set_mon_vals no callback set 2024-03-25T19:38:46.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.030+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fa5c4003fc0 con 0x7fa5cc1465a0 2024-03-25T19:38:46.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.031+0000 7fa5d1fd4640 10 monclient: get_auth_request con 0x7fa5b006dd00 auth_method 0 2024-03-25T19:38:46.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.031+0000 7fa5c27fc640 10 client.4394.objecter ms_handle_connect 0x7fa5b006dd00 2024-03-25T19:38:46.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fa5c4054b20 con 0x7fa5cc1465a0 2024-03-25T19:38:46.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 10 client.4394.objecter ms_dispatch 0x7fa5cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:46.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:46.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:46.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T19:38:46.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5c4050630 con 0x7fa5cc1465a0 2024-03-25T19:38:46.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:46.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.040+0000 7fa5c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:46.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.299+0000 7fa5d4238640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:46.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.299+0000 7fa5d4238640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:46.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.299+0000 7fa5d4238640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:46.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.299+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5cc0adf60 con 0x7fa5cc1465a0 2024-03-25T19:38:46.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.323+0000 7fa5c27fc640 1 -- 172.21.15.47:0/1040223438 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fa5c4052560 con 0x7fa5cc1465a0 2024-03-25T19:38:46.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.323+0000 7fa5c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:46.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.323+0000 7fa5c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:46.323 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:46.323 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:46.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 10 librados: watch_flush enter 2024-03-25T19:38:46.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 10 librados: watch_flush exit 2024-03-25T19:38:46.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T19:38:46.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 10 client.4394.objecter successfully canceled tick 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 >> v1:172.21.15.142:6800/1025302268 conn(0x7fa5b006dd00 legacy=0x7fa5b00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 10 monclient: shutdown 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.325+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 >> v1:172.21.15.47:6790/0 conn(0x7fa5cc1465a0 legacy=0x7fa5cc1e1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.326+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 shutdown_connections 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.326+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 >> 172.21.15.47:0/1040223438 conn(0x7fa5cc0b4a80 msgr2=0x7fa5cc070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.326+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 shutdown_connections 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.326+0000 7fa5d4238640 1 -- 172.21.15.47:0/1040223438 wait complete. 2024-03-25T19:38:46.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:46.326+0000 7fa5d4238640 1 librados: shutdown 2024-03-25T19:38:46.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:53.143 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:53.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.261+0000 7f345b84e640 10 monclient: get_monmap_and_config 2024-03-25T19:38:53.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.261+0000 7f345b84e640 10 monclient: build_initial_monmap 2024-03-25T19:38:53.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.261+0000 7f345b84e640 10 monclient: monmap: 2024-03-25T19:38:53.260 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:53.260 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:38:53.261862+0000 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:38:53.261862+0000 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:38:53.261 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:53.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.261+0000 7f345b84e640 1 Processor -- start 2024-03-25T19:38:53.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 1 -- start start 2024-03-25T19:38:53.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient: init 2024-03-25T19:38:53.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:53.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:53.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient(hunting): picked mon.noname-a con 0x7f34540a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient(hunting): picked mon.noname-c con 0x7f345413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient(hunting): picked mon.noname-b con 0x7f34541430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3454142520 con 0x7f34540a98f0 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540b28a0 con 0x7f345413f710 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540b2a30 con 0x7f34541430c0 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:53.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.262+0000 7f345b84e640 10 monclient(hunting): authenticate will time out at 5848.548918s 2024-03-25T19:38:53.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.273+0000 7f3458de9640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f345413f710 0x7f34541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:60764/0 (socket says 172.21.15.47:60764) 2024-03-25T19:38:53.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.273+0000 7f3458de9640 1 -- 172.21.15.47:0/1490136357 learned_addr learned my addr 172.21.15.47:0/1490136357 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:53.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.280+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265379319 0 0) 0x7f34540b28a0 con 0x7f345413f710 2024-03-25T19:38:53.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.280+0000 7f344bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.280+0000 7f344bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.280+0000 7f344bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.280+0000 7f344bfff640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:38:53.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.280+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3440003520 con 0x7f345413f710 2024-03-25T19:38:53.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.289+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726527637 0 0) 0x7f34540b2a30 con 0x7f34541430c0 2024-03-25T19:38:53.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.289+0000 7f344bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.289+0000 7f344bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.289+0000 7f344bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.289+0000 7f344bfff640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:38:53.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.289+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34540b28a0 con 0x7f34541430c0 2024-03-25T19:38:53.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689576921 0 0) 0x7f34540b28a0 con 0x7f34541430c0 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 >> v1:172.21.15.142:6789/0 conn(0x7f345413f710 legacy=0x7f34541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 >> v1:172.21.15.47:6789/0 conn(0x7f34540a98f0 legacy=0x7f34540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 1 monclient: found mon.noname-b 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540b44c0 con 0x7f34541430c0 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: _finish_auth 0 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f344c0019f0 con 0x7f34541430c0 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:53.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f344bfff640 10 monclient: dump: 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f345b84e640 5 monclient: authenticate success, global_id 4397 2024-03-25T19:38:53.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.303+0000 7f345b84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f344c001b50 con 0x7f34541430c0 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f344bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f344bfff640 1 -- 172.21.15.47:0/1490136357 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f344c001f20 con 0x7f34541430c0 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f345b84e640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f345b84e640 4 set_mon_vals no callback set 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f344bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f344bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:53.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.315+0000 7f344bfff640 10 monclient: dump: 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.316+0000 7f345b84e640 10 monclient: shutdown 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.316+0000 7f345b84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.316+0000 7f345b84e640 1 -- 172.21.15.47:0/1490136357 >> v1:172.21.15.47:6790/0 conn(0x7f34541430c0 legacy=0x7f345414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.316+0000 7f345b84e640 1 -- 172.21.15.47:0/1490136357 shutdown_connections 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.316+0000 7f345b84e640 1 -- 172.21.15.47:0/1490136357 >> 172.21.15.47:0/1490136357 conn(0x7f34540b7ea0 msgr2=0x7f34540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.317+0000 7f345b84e640 1 -- 172.21.15.47:0/1490136357 shutdown_connections 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.317+0000 7f345b84e640 1 -- 172.21.15.47:0/1490136357 wait complete. 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.317+0000 7f345b84e640 10 monclient: build_initial_monmap 2024-03-25T19:38:53.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.317+0000 7f345b84e640 10 monclient: monmap: 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 1 librados: starting msgr at 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 1 librados: starting objecter 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 1 Processor -- start 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 1 -- start start 2024-03-25T19:38:53.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 1 librados: setting wanted keys 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 1 librados: calling monclient init 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 10 monclient: init 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.318+0000 7f345b84e640 10 monclient(hunting): picked mon.noname-c con 0x7f34540a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 10 monclient(hunting): picked mon.noname-a con 0x7f345413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 10 monclient(hunting): picked mon.noname-b con 0x7f34541430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540ab960 con 0x7f345413f710 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34541db7b0 con 0x7f34541430c0 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34541db380 con 0x7f34540a98f0 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:53.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.319+0000 7f345b84e640 10 monclient(hunting): authenticate will time out at 5848.605219s 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.325+0000 7f34595ea640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f34540a98f0 0x7f34541dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44754/0 (socket says 172.21.15.47:44754) 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.325+0000 7f34595ea640 1 -- 172.21.15.47:0/1362164809 learned_addr learned my addr 172.21.15.47:0/1362164809 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.326+0000 7f3449ffb640 10 client.?.objecter ms_handle_connect 0x7f34540a98f0 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.326+0000 7f3449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:53.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.332+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244091104 0 0) 0x7f34541db380 con 0x7f34540a98f0 2024-03-25T19:38:53.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.332+0000 7f3449ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.332+0000 7f3449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.332+0000 7f3449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.332+0000 7f3449ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:38:53.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.332+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438004eb0 con 0x7f34540a98f0 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.341+0000 7f3449ffb640 10 client.?.objecter ms_handle_connect 0x7f34541430c0 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.341+0000 7f3449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.342+0000 7f3449ffb640 10 client.?.objecter ms_handle_connect 0x7f345413f710 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.342+0000 7f3449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1168333983 0 0) 0x7f3438004eb0 con 0x7f34540a98f0 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34541db380 con 0x7f34540a98f0 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f3450001a30 con 0x7f34540a98f0 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.347 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:53.348 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:53.348 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:53.348 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:53.348 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:53.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.347+0000 7f3449ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.361+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317614374 0 0) 0x7f34540ab960 con 0x7f345413f710 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.361+0000 7f3449ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.361+0000 7f3449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.361+0000 7f3449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.361+0000 7f3449ffb640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.361+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438004eb0 con 0x7f345413f710 2024-03-25T19:38:53.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.429+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407135335 0 0) 0x7f34541db7b0 con 0x7f34541430c0 2024-03-25T19:38:53.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.429+0000 7f3449ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.429+0000 7f3449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.429+0000 7f3449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.429+0000 7f3449ffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:38:53.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.429+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34540ab960 con 0x7f34541430c0 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372255470 0 0) 0x7f34541db380 con 0x7f34540a98f0 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 >> v1:172.21.15.142:6789/0 conn(0x7f34541430c0 legacy=0x7f3454137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 >> v1:172.21.15.47:6789/0 conn(0x7f345413f710 legacy=0x7f34540b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 1 monclient: found mon.c 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:53.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34541db510 con 0x7f34540a98f0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f3449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 5 monclient: authenticate success, global_id 4400 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 10 monclient: _renew_subs 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34540b07e0 con 0x7f34540a98f0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 10 monclient: _renew_subs 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34540b0f30 con 0x7f34540a98f0 2024-03-25T19:38:53.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.435+0000 7f345b84e640 1 librados: init done 2024-03-25T19:38:53.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.436+0000 7f345b84e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.436+0000 7f345b84e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.436+0000 7f345b84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:53.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.436+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34540b0b80 con 0x7f34540a98f0 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3450001cb0 con 0x7f34540a98f0 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f3449ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f3450003a60 con 0x7f34540a98f0 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f3449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f3449ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f3449ffb640 10 monclient: dump: 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.735 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:38:53.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.734+0000 7f344b7fe640 4 set_mon_vals no callback set 2024-03-25T19:38:53.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.735+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f3450003c40 con 0x7f34540a98f0 2024-03-25T19:38:53.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.735+0000 7f3458de9640 10 monclient: get_auth_request con 0x7f343806de60 auth_method 0 2024-03-25T19:38:53.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.736+0000 7f3449ffb640 10 client.4400.objecter ms_handle_connect 0x7f343806de60 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.742+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f34500548b0 con 0x7f34540a98f0 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.742+0000 7f3449ffb640 10 client.4400.objecter ms_dispatch 0x7f3454000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.742+0000 7f3449ffb640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.742+0000 7f3449ffb640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.743+0000 7f3449ffb640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.743+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34500503c0 con 0x7f34540a98f0 2024-03-25T19:38:53.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.743+0000 7f3449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.743+0000 7f3449ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:53.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.995+0000 7f345b84e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.995+0000 7f345b84e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.995+0000 7f345b84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:38:53.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:53.995+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34540627b0 con 0x7f34540a98f0 2024-03-25T19:38:54.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.030+0000 7f3449ffb640 1 -- 172.21.15.47:0/1362164809 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f34500522f0 con 0x7f34540a98f0 2024-03-25T19:38:54.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.030+0000 7f3449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:54.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.030+0000 7f3449ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:54.031 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:38:54.031 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 10 librados: watch_flush enter 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 10 librados: watch_flush exit 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 10 client.4400.objecter successfully canceled tick 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 >> v1:172.21.15.142:6800/1025302268 conn(0x7f343806de60 legacy=0x7f3438070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 10 monclient: shutdown 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:54.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 >> v1:172.21.15.47:6790/0 conn(0x7f34540a98f0 legacy=0x7f34541dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:54.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 shutdown_connections 2024-03-25T19:38:54.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 >> 172.21.15.47:0/1362164809 conn(0x7f34540b7ea0 msgr2=0x7f3454074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:54.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 shutdown_connections 2024-03-25T19:38:54.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 -- 172.21.15.47:0/1362164809 wait complete. 2024-03-25T19:38:54.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:38:54.032+0000 7f345b84e640 1 librados: shutdown 2024-03-25T19:38:54.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:57.809 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:35.045 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.815 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:35.050 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.842 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:35.078 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.867 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:04:35.057 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.870 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:35.106 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.871 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:04:35.061 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.878 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:35.113 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.917 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:04:35.107 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:57.922 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:35.158 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:00.852 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.968+0000 7f4ed38e0640 10 monclient: get_monmap_and_config 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.968+0000 7f4ed38e0640 10 monclient: build_initial_monmap 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.968+0000 7f4ed38e0640 10 monclient: monmap: 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:00.970525+0000 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:00.970525+0000 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:00.969 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:00.970 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:00.970 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:00.970 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:00.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.969+0000 7f4ed38e0640 1 Processor -- start 2024-03-25T19:39:00.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.969+0000 7f4ed38e0640 1 -- start start 2024-03-25T19:39:00.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.969+0000 7f4ed38e0640 10 monclient: init 2024-03-25T19:39:00.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:00.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4ecc0b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4ecc0b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4ecc14be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc0af940 con 0x7f4ecc0b4d60 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc0b0a70 con 0x7f4ecc14be90 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc0b0d50 con 0x7f4ecc0b1730 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.970+0000 7f4ed38e0640 10 monclient(hunting): authenticate will time out at 5856.257307s 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.979+0000 7f4ed167c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4ecc0b4d60 0x7f4ecc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49762/0 (socket says 172.21.15.47:49762) 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.979+0000 7f4ed167c640 1 -- 172.21.15.47:0/3346035549 learned_addr learned my addr 172.21.15.47:0/3346035549 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:00.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.991+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82111135 0 0) 0x7f4ecc0b0d50 con 0x7f4ecc0b1730 2024-03-25T19:39:00.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.991+0000 7f4ec3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.991+0000 7f4ec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.991+0000 7f4ec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.991+0000 7f4ec3fff640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:39:00.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.991+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb8003520 con 0x7f4ecc0b1730 2024-03-25T19:39:00.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.993+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413673401 0 0) 0x7f4ecc0af940 con 0x7f4ecc0b4d60 2024-03-25T19:39:00.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.993+0000 7f4ec3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.993+0000 7f4ec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.993+0000 7f4ec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.993+0000 7f4ec3fff640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:39:00.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:00.993+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ecc0b0d50 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2881871283 0 0) 0x7f4ecc0b0d50 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 >> v1:172.21.15.47:6790/0 conn(0x7f4ecc0b1730 legacy=0x7f4ecc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 >> v1:172.21.15.142:6789/0 conn(0x7f4ecc14be90 legacy=0x7f4ecc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 1 monclient: found mon.noname-a 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc0b1030 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: _finish_auth 0 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ec8001670 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ed38e0640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ed38e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:01.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ec3fff640 10 monclient: dump: 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:01.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.001+0000 7f4ed38e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:01.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ec80017d0 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ec3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ec3fff640 1 -- 172.21.15.47:0/3346035549 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ec8001c20 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ec3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ec3fff640 10 monclient: dump: 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:01.017 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:01.018 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:01.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ed38e0640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:01.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.016+0000 7f4ed38e0640 4 set_mon_vals no callback set 2024-03-25T19:39:01.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 10 monclient: shutdown 2024-03-25T19:39:01.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:01.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 1 -- 172.21.15.47:0/3346035549 >> v1:172.21.15.47:6789/0 conn(0x7f4ecc0b4d60 legacy=0x7f4ecc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 1 -- 172.21.15.47:0/3346035549 shutdown_connections 2024-03-25T19:39:01.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 1 -- 172.21.15.47:0/3346035549 >> 172.21.15.47:0/3346035549 conn(0x7f4ecc0617a0 msgr2=0x7f4ecc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:01.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 1 -- 172.21.15.47:0/3346035549 shutdown_connections 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.017+0000 7f4ed38e0640 1 -- 172.21.15.47:0/3346035549 wait complete. 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.018+0000 7f4ed38e0640 10 monclient: build_initial_monmap 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.018+0000 7f4ed38e0640 10 monclient: monmap: 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.018+0000 7f4ed38e0640 1 librados: starting msgr at 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.018+0000 7f4ed38e0640 1 librados: starting objecter 2024-03-25T19:39:01.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 Processor -- start 2024-03-25T19:39:01.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 -- start start 2024-03-25T19:39:01.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 librados: setting wanted keys 2024-03-25T19:39:01.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 librados: calling monclient init 2024-03-25T19:39:01.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient: init 2024-03-25T19:39:01.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4ecc0b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4ecc0b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4ecc14be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc1327e0 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc140810 con 0x7f4ecc0b1730 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc137810 con 0x7f4ecc14be90 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:01.022 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.019+0000 7f4ed38e0640 10 monclient(hunting): authenticate will time out at 5856.306704s 2024-03-25T19:39:01.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.037+0000 7f4ed1e7d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4ecc14be90 0x7f4ecc1320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49948/0 (socket says 172.21.15.47:49948) 2024-03-25T19:39:01.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.037+0000 7f4ed1e7d640 1 -- 172.21.15.47:0/1646092067 learned_addr learned my addr 172.21.15.47:0/1646092067 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:01.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.039+0000 7f4ec1ffb640 10 client.?.objecter ms_handle_connect 0x7f4ecc0b1730 2024-03-25T19:39:01.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.039+0000 7f4ec1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:01.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.041+0000 7f4ec1ffb640 10 client.?.objecter ms_handle_connect 0x7f4ecc14be90 2024-03-25T19:39:01.041 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.041+0000 7f4ec1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:01.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.051+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860851538 0 0) 0x7f4ecc140810 con 0x7f4ecc0b1730 2024-03-25T19:39:01.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.051+0000 7f4ec1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:01.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.051+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:01.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.051+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:01.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.051+0000 7f4ec1ffb640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:39:01.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.051+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0004eb0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.053+0000 7f4ec1ffb640 10 client.?.objecter ms_handle_connect 0x7f4ecc0b4d60 2024-03-25T19:39:01.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.053+0000 7f4ec1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:01.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.054+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739134254 0 0) 0x7f4ecc137810 con 0x7f4ecc14be90 2024-03-25T19:39:01.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.054+0000 7f4ec1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:01.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.054+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:01.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.054+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:01.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.054+0000 7f4ec1ffb640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:39:01.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.054+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ecc140810 con 0x7f4ecc14be90 2024-03-25T19:39:01.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3353082558 0 0) 0x7f4eb0004eb0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ecc137810 con 0x7f4ecc0b1730 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ec8001ba0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:01.070 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:01.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.069+0000 7f4ec1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:01.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.072+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045261386 0 0) 0x7f4ecc1327e0 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.072+0000 7f4ec1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:01.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.072+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:01.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.072+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:01.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.072+0000 7f4ec1ffb640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:39:01.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.072+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0004eb0 con 0x7f4ecc0b4d60 2024-03-25T19:39:01.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.079+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839665250 0 0) 0x7f4ecc140810 con 0x7f4ecc14be90 2024-03-25T19:39:01.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.079+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ecc1327e0 con 0x7f4ecc14be90 2024-03-25T19:39:01.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.079+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ec4001bd0 con 0x7f4ecc14be90 2024-03-25T19:39:01.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.079+0000 7f4ec1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:01.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.079+0000 7f4ec1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:01.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.079+0000 7f4ec1ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:01.080 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:01.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.089+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1893899895 0 0) 0x7f4ecc137810 con 0x7f4ecc0b1730 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.089+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 >> v1:172.21.15.47:6790/0 conn(0x7f4ecc14be90 legacy=0x7f4ecc1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.089+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 >> v1:172.21.15.47:6789/0 conn(0x7f4ecc0b4d60 legacy=0x7f4ecc1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.089+0000 7f4ec1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.089+0000 7f4ec1ffb640 1 monclient: found mon.b 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ec1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ec1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc1379a0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ec1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ec1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 10 monclient: _renew_subs 2024-03-25T19:39:01.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ecc140380 con 0x7f4ecc0b1730 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 10 monclient: _renew_subs 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc138290 con 0x7f4ecc0b1730 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.090+0000 7f4ed38e0640 1 librados: init done 2024-03-25T19:39:01.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.091+0000 7f4ed38e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:01.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.091+0000 7f4ed38e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:01.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.091+0000 7f4ed38e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:01.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.091+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ecc0627b0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ec8004a90 con 0x7f4ecc0b1730 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ec8004da0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec1ffb640 10 monclient: dump: 2024-03-25T19:39:01.392 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:01.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.391+0000 7f4ec37fe640 4 set_mon_vals no callback set 2024-03-25T19:39:01.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.392+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f4ec801ad70 con 0x7f4ecc0b1730 2024-03-25T19:39:01.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.393+0000 7f4ed0e7b640 10 monclient: get_auth_request con 0x7f4eb006df70 auth_method 0 2024-03-25T19:39:01.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.393+0000 7f4ec1ffb640 10 client.4420.objecter ms_handle_connect 0x7f4eb006df70 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.397+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f4ec8054920 con 0x7f4ecc0b1730 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.397+0000 7f4ec1ffb640 10 client.4420.objecter ms_dispatch 0x7f4ecc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.397+0000 7f4ec1ffb640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.397+0000 7f4ec1ffb640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.397+0000 7f4ec1ffb640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.398+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ec8050260 con 0x7f4ecc0b1730 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.398+0000 7f4ec1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:01.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.398+0000 7f4ec1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:01.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.621+0000 7f4ed38e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.621+0000 7f4ed38e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.621+0000 7f4ed38e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:01.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.621+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ecc1409a0 con 0x7f4ecc0b1730 2024-03-25T19:39:01.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.672+0000 7f4ec1ffb640 1 -- 172.21.15.47:0/1646092067 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f4ec8052360 con 0x7f4ecc0b1730 2024-03-25T19:39:01.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.672+0000 7f4ec1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.672+0000 7f4ec1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:01.673 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:01.673 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 10 librados: watch_flush enter 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 10 librados: watch_flush exit 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 10 client.4420.objecter successfully canceled tick 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 >> v1:172.21.15.142:6800/1025302268 conn(0x7f4eb006df70 legacy=0x7f4eb0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 10 monclient: shutdown 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 >> v1:172.21.15.142:6789/0 conn(0x7f4ecc0b1730 legacy=0x7f4ecc1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 shutdown_connections 2024-03-25T19:39:01.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 >> 172.21.15.47:0/1646092067 conn(0x7f4ecc0617a0 msgr2=0x7f4ecc0b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:01.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 shutdown_connections 2024-03-25T19:39:01.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 -- 172.21.15.47:0/1646092067 wait complete. 2024-03-25T19:39:01.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:01.675+0000 7f4ed38e0640 1 librados: shutdown 2024-03-25T19:39:01.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:08.494 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.607+0000 7f1dc0752640 10 monclient: get_monmap_and_config 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.607+0000 7f1dc0752640 10 monclient: build_initial_monmap 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.607+0000 7f1dc0752640 10 monclient: monmap: 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:08.608585+0000 2024-03-25T19:39:08.607 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:08.608585+0000 2024-03-25T19:39:08.608 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:08.608 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:08.608 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:08.608 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:08.608 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:08.608 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:08.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.608+0000 7f1dc0752640 1 Processor -- start 2024-03-25T19:39:08.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.608+0000 7f1dc0752640 1 -- start start 2024-03-25T19:39:08.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.608+0000 7f1dc0752640 10 monclient: init 2024-03-25T19:39:08.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:08.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient(hunting): picked mon.noname-b con 0x7f1db80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient(hunting): picked mon.noname-c con 0x7f1db813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient(hunting): picked mon.noname-a con 0x7f1db81430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db8142520 con 0x7f1db81430c0 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db80b28a0 con 0x7f1db813f710 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db80b2a30 con 0x7f1db80a98f0 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:08.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.609+0000 7f1dc0752640 10 monclient(hunting): authenticate will time out at 5863.895684s 2024-03-25T19:39:08.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.615+0000 7f1dbe4ee640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1db80a98f0 0x7f1db80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48216/0 (socket says 172.21.15.47:48216) 2024-03-25T19:39:08.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.615+0000 7f1dbe4ee640 1 -- 172.21.15.47:0/11991195 learned_addr learned my addr 172.21.15.47:0/11991195 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:08.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.625+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661371689 0 0) 0x7f1db80b2a30 con 0x7f1db80a98f0 2024-03-25T19:39:08.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.625+0000 7f1dbd4ec640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.625+0000 7f1dbd4ec640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.625+0000 7f1dbd4ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.625+0000 7f1dbd4ec640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:39:08.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.625+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da4003520 con 0x7f1db80a98f0 2024-03-25T19:39:08.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499306384 0 0) 0x7f1db80b28a0 con 0x7f1db813f710 2024-03-25T19:39:08.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:39:08.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db80b2a30 con 0x7f1db813f710 2024-03-25T19:39:08.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 884569073 0 0) 0x7f1da4003520 con 0x7f1db80a98f0 2024-03-25T19:39:08.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 >> v1:172.21.15.142:6789/0 conn(0x7f1db813f710 legacy=0x7f1db81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.632+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 >> v1:172.21.15.47:6789/0 conn(0x7f1db81430c0 legacy=0x7f1db814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: _finish_hunting 0 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 1 monclient: found mon.noname-b 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db80b44c0 con 0x7f1db80a98f0 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: _finish_auth 0 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: _check_auth_tickets 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1db4001690 con 0x7f1db80a98f0 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dbd4ec640 10 monclient: dump: 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.633 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dc0752640 5 monclient: authenticate success, global_id 4409 2024-03-25T19:39:08.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.633+0000 7f1dc0752640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db40017f0 con 0x7f1db80a98f0 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dbd4ec640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dbd4ec640 1 -- 172.21.15.47:0/11991195 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1db4001c00 con 0x7f1db80a98f0 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dc0752640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dc0752640 4 set_mon_vals no callback set 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dbd4ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dbd4ec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.640+0000 7f1dbd4ec640 10 monclient: dump: 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.640 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.641 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:08.641 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:08.641 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:08.641 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:08.641 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:08.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 10 monclient: shutdown 2024-03-25T19:39:08.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:08.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 1 -- 172.21.15.47:0/11991195 >> v1:172.21.15.47:6790/0 conn(0x7f1db80a98f0 legacy=0x7f1db80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 1 -- 172.21.15.47:0/11991195 shutdown_connections 2024-03-25T19:39:08.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 1 -- 172.21.15.47:0/11991195 >> 172.21.15.47:0/11991195 conn(0x7f1db80b7ea0 msgr2=0x7f1db80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:08.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 1 -- 172.21.15.47:0/11991195 shutdown_connections 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.641+0000 7f1dc0752640 1 -- 172.21.15.47:0/11991195 wait complete. 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.642+0000 7f1dc0752640 10 monclient: build_initial_monmap 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.642+0000 7f1dc0752640 10 monclient: monmap: 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 librados: starting msgr at 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 librados: starting objecter 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 Processor -- start 2024-03-25T19:39:08.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 -- start start 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 librados: setting wanted keys 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 librados: calling monclient init 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient: init 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient(hunting): picked mon.noname-c con 0x7f1db80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:08.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient(hunting): picked mon.noname-a con 0x7f1db813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:08.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient(hunting): picked mon.noname-b con 0x7f1db81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:08.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db813be70 con 0x7f1db813f710 2024-03-25T19:39:08.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db8131660 con 0x7f1db81430c0 2024-03-25T19:39:08.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db81317f0 con 0x7f1db80a98f0 2024-03-25T19:39:08.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:08.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.643+0000 7f1dc0752640 10 monclient(hunting): authenticate will time out at 5863.929803s 2024-03-25T19:39:08.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.662+0000 7f1dbe4ee640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1db80a98f0 0x7f1db8130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48264/0 (socket says 172.21.15.47:48264) 2024-03-25T19:39:08.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.662+0000 7f1dbe4ee640 1 -- 172.21.15.47:0/392998019 learned_addr learned my addr 172.21.15.47:0/392998019 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:08.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.663+0000 7f1daeffd640 10 client.?.objecter ms_handle_connect 0x7f1db80a98f0 2024-03-25T19:39:08.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.663+0000 7f1daeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.671+0000 7f1daeffd640 10 client.?.objecter ms_handle_connect 0x7f1db813f710 2024-03-25T19:39:08.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.671+0000 7f1daeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.672+0000 7f1daeffd640 10 client.?.objecter ms_handle_connect 0x7f1db81430c0 2024-03-25T19:39:08.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.672+0000 7f1daeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.677+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16496714 0 0) 0x7f1db81317f0 con 0x7f1db80a98f0 2024-03-25T19:39:08.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.677+0000 7f1daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.677+0000 7f1daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.677+0000 7f1daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.677+0000 7f1daeffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:39:08.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.677+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c004eb0 con 0x7f1db80a98f0 2024-03-25T19:39:08.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.682+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001927176 0 0) 0x7f1db813be70 con 0x7f1db813f710 2024-03-25T19:39:08.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.682+0000 7f1daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.682+0000 7f1daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.682+0000 7f1daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.682+0000 7f1daeffd640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:39:08.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.682+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db81317f0 con 0x7f1db813f710 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485283451 0 0) 0x7f1d9c004eb0 con 0x7f1db80a98f0 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1db813be70 con 0x7f1db80a98f0 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1db4001c40 con 0x7f1db80a98f0 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:08.688 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:08.689 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:08.689 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.688+0000 7f1daeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:08.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.693+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229472081 0 0) 0x7f1db8131660 con 0x7f1db81430c0 2024-03-25T19:39:08.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.693+0000 7f1daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.693+0000 7f1daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.693+0000 7f1daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.693+0000 7f1daeffd640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:39:08.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.693+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c004eb0 con 0x7f1db81430c0 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1552255554 0 0) 0x7f1db813be70 con 0x7f1db80a98f0 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 >> v1:172.21.15.142:6789/0 conn(0x7f1db81430c0 legacy=0x7f1db81e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 >> v1:172.21.15.47:6789/0 conn(0x7f1db813f710 legacy=0x7f1db81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 1 monclient: found mon.c 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db8131980 con 0x7f1db80a98f0 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 5 monclient: authenticate success, global_id 4412 2024-03-25T19:39:08.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 10 monclient: _renew_subs 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db8135c60 con 0x7f1db80a98f0 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 10 monclient: _renew_subs 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db80b7770 con 0x7f1db80a98f0 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.695+0000 7f1dc0752640 1 librados: init done 2024-03-25T19:39:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.696+0000 7f1dc0752640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.696+0000 7f1dc0752640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.696+0000 7f1dc0752640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:39:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.696+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db81371d0 con 0x7f1db80a98f0 2024-03-25T19:39:08.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db4004a90 con 0x7f1db80a98f0 2024-03-25T19:39:08.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:08.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1db4004de0 con 0x7f1db80a98f0 2024-03-25T19:39:08.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 10 monclient: dump: 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:08.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1dbcceb640 4 set_mon_vals no callback set 2024-03-25T19:39:08.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.995+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f1db4005000 con 0x7f1db80a98f0 2024-03-25T19:39:08.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.996+0000 7f1dbdced640 10 monclient: get_auth_request con 0x7f1d9c06de90 auth_method 0 2024-03-25T19:39:08.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:08.996+0000 7f1daeffd640 10 client.4412.objecter ms_handle_connect 0x7f1d9c06de90 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.003+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f1db40549b0 con 0x7f1db80a98f0 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.003+0000 7f1daeffd640 10 client.4412.objecter ms_dispatch 0x7f1db8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.003+0000 7f1daeffd640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.003+0000 7f1daeffd640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.003+0000 7f1daeffd640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.003+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1db40504c0 con 0x7f1db80a98f0 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.004+0000 7f1daeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:09.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.004+0000 7f1daeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:09.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.262+0000 7f1dc0752640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:09.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.262+0000 7f1dc0752640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:09.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.262+0000 7f1dc0752640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:39:09.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.262+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1db80627b0 con 0x7f1db80a98f0 2024-03-25T19:39:09.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.291+0000 7f1daeffd640 1 -- 172.21.15.47:0/392998019 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f1db40523f0 con 0x7f1db80a98f0 2024-03-25T19:39:09.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.291+0000 7f1daeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:09.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.291+0000 7f1daeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:09.291 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:09.291 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 10 librados: watch_flush enter 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 10 librados: watch_flush exit 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 10 client.4412.objecter successfully canceled tick 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 >> v1:172.21.15.142:6800/1025302268 conn(0x7f1d9c06de90 legacy=0x7f1d9c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 10 monclient: shutdown 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 >> v1:172.21.15.47:6790/0 conn(0x7f1db80a98f0 legacy=0x7f1db8130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:09.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 shutdown_connections 2024-03-25T19:39:09.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 >> 172.21.15.47:0/392998019 conn(0x7f1db80b7ea0 msgr2=0x7f1db8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:09.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 shutdown_connections 2024-03-25T19:39:09.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 -- 172.21.15.47:0/392998019 wait complete. 2024-03-25T19:39:09.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:09.293+0000 7f1dc0752640 1 librados: shutdown 2024-03-25T19:39:09.304 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:16.110 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.225+0000 7f5464dd7640 10 monclient: get_monmap_and_config 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.225+0000 7f5464dd7640 10 monclient: build_initial_monmap 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.225+0000 7f5464dd7640 10 monclient: monmap: 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:16.226542+0000 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:16.226542+0000 2024-03-25T19:39:16.225 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:16.226 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:16.226 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:16.226 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:16.226 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:16.226 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:16.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.226+0000 7f5464dd7640 1 Processor -- start 2024-03-25T19:39:16.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.226+0000 7f5464dd7640 1 -- start start 2024-03-25T19:39:16.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.226+0000 7f5464dd7640 10 monclient: init 2024-03-25T19:39:16.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:16.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:16.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient(hunting): picked mon.noname-b con 0x7f54600a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f546013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f54601430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5460142520 con 0x7f54601430c0 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600b28a0 con 0x7f546013f710 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600b2a30 con 0x7f54600a98f0 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.227+0000 7f5464dd7640 10 monclient(hunting): authenticate will time out at 5871.513191s 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.228+0000 7f545ed9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f54601430c0 0x7f546014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38956/0 (socket says 172.21.15.47:38956) 2024-03-25T19:39:16.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.228+0000 7f545ed9d640 1 -- 172.21.15.47:0/3470965566 learned_addr learned my addr 172.21.15.47:0/3470965566 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:16.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.234+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335639283 0 0) 0x7f5460142520 con 0x7f54601430c0 2024-03-25T19:39:16.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.234+0000 7f545dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.234+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.234+0000 7f545dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.234+0000 7f545dd9b640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:39:16.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.234+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f544c003520 con 0x7f54601430c0 2024-03-25T19:39:16.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.247+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928069820 0 0) 0x7f54600b28a0 con 0x7f546013f710 2024-03-25T19:39:16.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.247+0000 7f545dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.247+0000 7f545dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.247+0000 7f545dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.247+0000 7f545dd9b640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:39:16.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.247+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5460142520 con 0x7f546013f710 2024-03-25T19:39:16.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360299790 0 0) 0x7f544c003520 con 0x7f54601430c0 2024-03-25T19:39:16.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 >> v1:172.21.15.47:6790/0 conn(0x7f54600a98f0 legacy=0x7f54600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 >> v1:172.21.15.142:6789/0 conn(0x7f546013f710 legacy=0x7f54601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:39:16.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600b44c0 con 0x7f54601430c0 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5458001a10 con 0x7f54601430c0 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.248+0000 7f545dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.249+0000 7f545dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.249+0000 7f545dd9b640 10 monclient: dump: 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:16.249 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:16.250 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:16.250 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:16.250 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:16.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.249+0000 7f5464dd7640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:39:16.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.249+0000 7f5464dd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:16.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5458001b70 con 0x7f54601430c0 2024-03-25T19:39:16.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f545dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f545dd9b640 1 -- 172.21.15.47:0/3470965566 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5458001f80 con 0x7f54601430c0 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f5464dd7640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f5464dd7640 4 set_mon_vals no callback set 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f545dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f545dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.253+0000 7f545dd9b640 10 monclient: dump: 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:16.253 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:16.254 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:16.254 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:16.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 10 monclient: shutdown 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 1 -- 172.21.15.47:0/3470965566 >> v1:172.21.15.47:6789/0 conn(0x7f54601430c0 legacy=0x7f546014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 1 -- 172.21.15.47:0/3470965566 shutdown_connections 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 1 -- 172.21.15.47:0/3470965566 >> 172.21.15.47:0/3470965566 conn(0x7f54600b7ea0 msgr2=0x7f54600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 1 -- 172.21.15.47:0/3470965566 shutdown_connections 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.254+0000 7f5464dd7640 1 -- 172.21.15.47:0/3470965566 wait complete. 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.255+0000 7f5464dd7640 10 monclient: build_initial_monmap 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.255+0000 7f5464dd7640 10 monclient: monmap: 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:16.255 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:16.256 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:16.256 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:16.256 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:16.256 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:16.256 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:16.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 1 librados: starting msgr at 2024-03-25T19:39:16.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 1 librados: starting objecter 2024-03-25T19:39:16.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 1 Processor -- start 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 1 -- start start 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 1 librados: setting wanted keys 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 1 librados: calling monclient init 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.256+0000 7f5464dd7640 10 monclient: init 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f54600a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f546013f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 10 monclient(hunting): picked mon.noname-b con 0x7f54601430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600abb00 con 0x7f54600a98f0 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.257+0000 7f5464dd7640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600abde0 con 0x7f54601430c0 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.258+0000 7f5464dd7640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5460138e20 con 0x7f546013f710 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.258+0000 7f5464dd7640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:16.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.258+0000 7f5464dd7640 10 monclient(hunting): authenticate will time out at 5871.543968s 2024-03-25T19:39:16.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.262+0000 7f545ed9d640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f54601430c0 0x7f54600ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:46500/0 (socket says 172.21.15.47:46500) 2024-03-25T19:39:16.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.263+0000 7f545ed9d640 1 -- 172.21.15.47:0/1054878404 learned_addr learned my addr 172.21.15.47:0/1054878404 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:16.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.266+0000 7f54577fe640 10 client.?.objecter ms_handle_connect 0x7f54601430c0 2024-03-25T19:39:16.266 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.266+0000 7f54577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:16.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.274+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171320996 0 0) 0x7f54600abde0 con 0x7f54601430c0 2024-03-25T19:39:16.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.274+0000 7f54577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.274+0000 7f54577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.274+0000 7f54577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.274+0000 7f54577fe640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:39:16.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.274+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5440004eb0 con 0x7f54601430c0 2024-03-25T19:39:16.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.279+0000 7f54577fe640 10 client.?.objecter ms_handle_connect 0x7f546013f710 2024-03-25T19:39:16.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.279+0000 7f54577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:16.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1314321603 0 0) 0x7f5440004eb0 con 0x7f54601430c0 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54600abde0 con 0x7f54601430c0 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5458001ec0 con 0x7f54601430c0 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 10 monclient(hunting): dump: 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:16.287 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:16.288 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:16.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.287+0000 7f54577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:16.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.289+0000 7f54577fe640 10 client.?.objecter ms_handle_connect 0x7f54600a98f0 2024-03-25T19:39:16.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.289+0000 7f54577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:16.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.301+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863969836 0 0) 0x7f5460138e20 con 0x7f546013f710 2024-03-25T19:39:16.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.301+0000 7f54577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.301+0000 7f54577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.301+0000 7f54577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.301+0000 7f54577fe640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:39:16.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.301+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5440004eb0 con 0x7f546013f710 2024-03-25T19:39:16.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.304+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917150275 0 0) 0x7f54600abb00 con 0x7f54600a98f0 2024-03-25T19:39:16.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.304+0000 7f54577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.304+0000 7f54577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.304+0000 7f54577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.304+0000 7f54577fe640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:39:16.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.304+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5460138e20 con 0x7f54600a98f0 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2405393730 0 0) 0x7f54600abde0 con 0x7f54601430c0 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 >> v1:172.21.15.47:6790/0 conn(0x7f546013f710 legacy=0x7f5460136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 >> v1:172.21.15.47:6789/0 conn(0x7f54600a98f0 legacy=0x7f54601385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 1 monclient: found mon.b 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5460137b00 con 0x7f54601430c0 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:16.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f54577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 10 monclient: _renew_subs 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54601e34b0 con 0x7f54601430c0 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 10 monclient: _renew_subs 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54601e3820 con 0x7f54601430c0 2024-03-25T19:39:16.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.310+0000 7f5464dd7640 1 librados: init done 2024-03-25T19:39:16.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.311+0000 7f5464dd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.311+0000 7f5464dd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.311+0000 7f5464dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:16.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.311+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54600627b0 con 0x7f54601430c0 2024-03-25T19:39:16.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5458004e10 con 0x7f54601430c0 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f54577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5458002020 con 0x7f54601430c0 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f54577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f54577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f54577fe640 10 monclient: dump: 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:16.608 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:16.609 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:16.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.608+0000 7f545d59a640 4 set_mon_vals no callback set 2024-03-25T19:39:16.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.609+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f545801af90 con 0x7f54601430c0 2024-03-25T19:39:16.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.609+0000 7f545e59c640 10 monclient: get_auth_request con 0x7f544006de10 auth_method 0 2024-03-25T19:39:16.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.609+0000 7f54577fe640 10 client.4432.objecter ms_handle_connect 0x7f544006de10 2024-03-25T19:39:16.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f5458054bb0 con 0x7f54601430c0 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 10 client.4432.objecter ms_dispatch 0x7f5460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54580504f0 con 0x7f54601430c0 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.614+0000 7f54577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:16.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.851+0000 7f5464dd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.851+0000 7f5464dd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.851+0000 7f5464dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:16.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.851+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54600b1760 con 0x7f54601430c0 2024-03-25T19:39:16.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.877+0000 7f54577fe640 1 -- 172.21.15.47:0/1054878404 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f54580525f0 con 0x7f54601430c0 2024-03-25T19:39:16.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.877+0000 7f54577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.877+0000 7f54577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:16.877 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:16.877 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:16.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.879+0000 7f5464dd7640 10 librados: watch_flush enter 2024-03-25T19:39:16.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.879+0000 7f5464dd7640 10 librados: watch_flush exit 2024-03-25T19:39:16.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.879+0000 7f5464dd7640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T19:39:16.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.879+0000 7f5464dd7640 10 client.4432.objecter successfully canceled tick 2024-03-25T19:39:16.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.879+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 >> v1:172.21.15.142:6800/1025302268 conn(0x7f544006de10 legacy=0x7f54400702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 10 monclient: shutdown 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 >> v1:172.21.15.142:6789/0 conn(0x7f54601430c0 legacy=0x7f54600ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 shutdown_connections 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 >> 172.21.15.47:0/1054878404 conn(0x7f54600b7ea0 msgr2=0x7f546014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 shutdown_connections 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 1 -- 172.21.15.47:0/1054878404 wait complete. 2024-03-25T19:39:16.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:16.880+0000 7f5464dd7640 1 librados: shutdown 2024-03-25T19:39:16.892 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:17.880 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:55.115 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.885 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:55.121 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.932 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:04:55.122 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.932 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:55.167 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.943 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:55.179 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.946 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:04:55.136 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.948 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:55.184 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.973 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:04:55.164 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.981 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:55.217 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:23.698 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.810+0000 7fc7ba925640 10 monclient: get_monmap_and_config 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.810+0000 7fc7ba925640 10 monclient: build_initial_monmap 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.811+0000 7fc7ba925640 10 monclient: monmap: 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:23.811782+0000 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:23.811782+0000 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.811+0000 7fc7ba925640 1 Processor -- start 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.811+0000 7fc7ba925640 1 -- start start 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.811+0000 7fc7ba925640 10 monclient: init 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b40a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b41430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b4142520 con 0x7fc7b413f710 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40b28a0 con 0x7fc7b40a98f0 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40b2a30 con 0x7fc7b41430c0 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.812+0000 7fc7ba925640 10 monclient(hunting): authenticate will time out at 5879.098480s 2024-03-25T19:39:23.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.825+0000 7fc7b3fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc7b40a98f0 0x7fc7b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:42122/0 (socket says 172.21.15.47:42122) 2024-03-25T19:39:23.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.825+0000 7fc7b3fff640 1 -- 172.21.15.47:0/267990079 learned_addr learned my addr 172.21.15.47:0/267990079 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:23.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.834+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020367039 0 0) 0x7fc7b4142520 con 0x7fc7b413f710 2024-03-25T19:39:23.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.834+0000 7fc7b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.834+0000 7fc7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.834+0000 7fc7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.834+0000 7fc7b2ffd640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:39:23.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.834+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc79c003520 con 0x7fc7b413f710 2024-03-25T19:39:23.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.838+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885171991 0 0) 0x7fc7b40b28a0 con 0x7fc7b40a98f0 2024-03-25T19:39:23.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.838+0000 7fc7b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.838+0000 7fc7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.838+0000 7fc7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.838+0000 7fc7b2ffd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:39:23.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.838+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b4142520 con 0x7fc7b40a98f0 2024-03-25T19:39:23.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.842+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810706353 0 0) 0x7fc7b40b2a30 con 0x7fc7b41430c0 2024-03-25T19:39:23.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.842+0000 7fc7b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.842+0000 7fc7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.842+0000 7fc7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.842+0000 7fc7b2ffd640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:39:23.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.842+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b40b28a0 con 0x7fc7b41430c0 2024-03-25T19:39:23.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973126779 0 0) 0x7fc7b40b28a0 con 0x7fc7b41430c0 2024-03-25T19:39:23.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 >> v1:172.21.15.142:6789/0 conn(0x7fc7b40a98f0 legacy=0x7fc7b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 >> v1:172.21.15.47:6789/0 conn(0x7fc7b413f710 legacy=0x7fc7b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b40b44c0 con 0x7fc7b41430c0 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc7ac001930 con 0x7fc7b41430c0 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7ba925640 5 monclient: authenticate success, global_id 4421 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7ba925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7b2ffd640 10 monclient: dump: 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:23.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.853+0000 7fc7ba925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7ac001a90 con 0x7fc7b41430c0 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7b2ffd640 1 -- 172.21.15.47:0/267990079 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc7ac001ee0 con 0x7fc7b41430c0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7b2ffd640 10 monclient: dump: 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7ba925640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:23.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.863+0000 7fc7ba925640 4 set_mon_vals no callback set 2024-03-25T19:39:23.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 10 monclient: shutdown 2024-03-25T19:39:23.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:23.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 1 -- 172.21.15.47:0/267990079 >> v1:172.21.15.47:6790/0 conn(0x7fc7b41430c0 legacy=0x7fc7b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 1 -- 172.21.15.47:0/267990079 shutdown_connections 2024-03-25T19:39:23.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 1 -- 172.21.15.47:0/267990079 >> 172.21.15.47:0/267990079 conn(0x7fc7b40b7ea0 msgr2=0x7fc7b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 1 -- 172.21.15.47:0/267990079 shutdown_connections 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.864+0000 7fc7ba925640 1 -- 172.21.15.47:0/267990079 wait complete. 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.865+0000 7fc7ba925640 10 monclient: build_initial_monmap 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.865+0000 7fc7ba925640 10 monclient: monmap: 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:23.866 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.865+0000 7fc7ba925640 1 librados: starting msgr at 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.865+0000 7fc7ba925640 1 librados: starting objecter 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.865+0000 7fc7ba925640 1 Processor -- start 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 1 -- start start 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 1 librados: setting wanted keys 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 1 librados: calling monclient init 2024-03-25T19:39:23.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient: init 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b40a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b41430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b406fc00 con 0x7fc7b413f710 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b4070430 con 0x7fc7b41430c0 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b406fe70 con 0x7fc7b40a98f0 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:23.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.866+0000 7fc7ba925640 10 monclient(hunting): authenticate will time out at 5879.152833s 2024-03-25T19:39:23.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.874+0000 7fc7b8ec2640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc7b41430c0 0x7fc7b41e64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:42160/0 (socket says 172.21.15.47:42160) 2024-03-25T19:39:23.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.874+0000 7fc7b8ec2640 1 -- 172.21.15.47:0/1727145532 learned_addr learned my addr 172.21.15.47:0/1727145532 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:23.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.874+0000 7fc7b0ff9640 10 client.?.objecter ms_handle_connect 0x7fc7b41430c0 2024-03-25T19:39:23.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.874+0000 7fc7b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.879+0000 7fc7b0ff9640 10 client.?.objecter ms_handle_connect 0x7fc7b413f710 2024-03-25T19:39:23.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.879+0000 7fc7b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.885+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508771316 0 0) 0x7fc7b406fc00 con 0x7fc7b413f710 2024-03-25T19:39:23.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.885+0000 7fc7b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.885+0000 7fc7b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.885+0000 7fc7b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.885+0000 7fc7b0ff9640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:39:23.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.885+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0004eb0 con 0x7fc7b413f710 2024-03-25T19:39:23.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.888+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353588297 0 0) 0x7fc7b4070430 con 0x7fc7b41430c0 2024-03-25T19:39:23.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.888+0000 7fc7b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.888+0000 7fc7b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.888+0000 7fc7b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.888+0000 7fc7b0ff9640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:39:23.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.888+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b406fc00 con 0x7fc7b41430c0 2024-03-25T19:39:23.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.899+0000 7fc7b0ff9640 10 client.?.objecter ms_handle_connect 0x7fc7b40a98f0 2024-03-25T19:39:23.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.899+0000 7fc7b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188262951 0 0) 0x7fc7b406fc00 con 0x7fc7b41430c0 2024-03-25T19:39:23.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7b4070430 con 0x7fc7b41430c0 2024-03-25T19:39:23.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc7ac001ea0 con 0x7fc7b41430c0 2024-03-25T19:39:23.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:23.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.905+0000 7fc7b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.908+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122636967 0 0) 0x7fc7a0004eb0 con 0x7fc7b413f710 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.908+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7b406fc00 con 0x7fc7b413f710 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.908+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc7a8001950 con 0x7fc7b413f710 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.908+0000 7fc7b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.908+0000 7fc7b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.908+0000 7fc7b0ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:23.908 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.909 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:23.909 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:23.909 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:23.909 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:23.909 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:23.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1230206685 0 0) 0x7fc7b406fc00 con 0x7fc7b413f710 2024-03-25T19:39:23.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 >> v1:172.21.15.47:6790/0 conn(0x7fc7b40a98f0 legacy=0x7fc7b406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 >> v1:172.21.15.142:6789/0 conn(0x7fc7b41430c0 legacy=0x7fc7b41e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 1 monclient: found mon.a 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b4070150 con 0x7fc7b413f710 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.917+0000 7fc7ba925640 5 monclient: authenticate success, global_id 4446 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 10 monclient: _renew_subs 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b406f080 con 0x7fc7b413f710 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:23.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 10 monclient: _renew_subs 2024-03-25T19:39:23.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:23.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b4070aa0 con 0x7fc7b413f710 2024-03-25T19:39:23.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.918+0000 7fc7ba925640 1 librados: init done 2024-03-25T19:39:23.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.919+0000 7fc7ba925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:23.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.919+0000 7fc7ba925640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:23.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.919+0000 7fc7ba925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:23.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:23.919+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b406f270 con 0x7fc7b413f710 2024-03-25T19:39:24.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.253+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7a80035f0 con 0x7fc7b413f710 2024-03-25T19:39:24.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:24.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc7a8003900 con 0x7fc7b413f710 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b0ff9640 10 monclient: dump: 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:24.254 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:24.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b27fc640 4 set_mon_vals no callback set 2024-03-25T19:39:24.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.254+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fc7a80198d0 con 0x7fc7b413f710 2024-03-25T19:39:24.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.255+0000 7fc7b3fff640 10 monclient: get_auth_request con 0x7fc7a006de90 auth_method 0 2024-03-25T19:39:24.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.255+0000 7fc7b0ff9640 10 client.4446.objecter ms_handle_connect 0x7fc7a006de90 2024-03-25T19:39:24.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.260+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fc7a8053560 con 0x7fc7b413f710 2024-03-25T19:39:24.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.260+0000 7fc7b0ff9640 10 client.4446.objecter ms_dispatch 0x7fc7b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:24.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.260+0000 7fc7b0ff9640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:24.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.260+0000 7fc7b0ff9640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:24.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.260+0000 7fc7b0ff9640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-25T19:39:24.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.261+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7a804eea0 con 0x7fc7b413f710 2024-03-25T19:39:24.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.261+0000 7fc7b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:24.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.261+0000 7fc7b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:24.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.488+0000 7fc7ba925640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.488+0000 7fc7ba925640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.488+0000 7fc7ba925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:24.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.488+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7b40627b0 con 0x7fc7b413f710 2024-03-25T19:39:24.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.510+0000 7fc7b0ff9640 1 -- 172.21.15.47:0/1727145532 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fc7a8050fa0 con 0x7fc7b413f710 2024-03-25T19:39:24.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.510+0000 7fc7b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.510+0000 7fc7b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:24.510 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:24.510 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 10 librados: watch_flush enter 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 10 librados: watch_flush exit 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 10 client.4446.objecter successfully canceled tick 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 >> v1:172.21.15.142:6800/1025302268 conn(0x7fc7a006de90 legacy=0x7fc7a0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 10 monclient: shutdown 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:24.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 >> v1:172.21.15.47:6789/0 conn(0x7fc7b413f710 legacy=0x7fc7b414bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:24.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 shutdown_connections 2024-03-25T19:39:24.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 >> 172.21.15.47:0/1727145532 conn(0x7fc7b40b7ea0 msgr2=0x7fc7b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:24.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 shutdown_connections 2024-03-25T19:39:24.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 -- 172.21.15.47:0/1727145532 wait complete. 2024-03-25T19:39:24.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:24.513+0000 7fc7ba925640 1 librados: shutdown 2024-03-25T19:39:24.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:31.331 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:31.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.431+0000 7fc52bbaf640 10 monclient: get_monmap_and_config 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.431+0000 7fc52bbaf640 10 monclient: build_initial_monmap 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.431+0000 7fc52bbaf640 10 monclient: monmap: 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:31.432385+0000 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:31.432385+0000 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:31.431 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.432+0000 7fc52bbaf640 1 Processor -- start 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.432+0000 7fc52bbaf640 1 -- start start 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.432+0000 7fc52bbaf640 10 monclient: init 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient(hunting): picked mon.noname-b con 0x7fc5240a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient(hunting): picked mon.noname-a con 0x7fc52413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient(hunting): picked mon.noname-c con 0x7fc5241430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc524142520 con 0x7fc52413f710 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240b28a0 con 0x7fc5241430c0 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240b2a30 con 0x7fc5240a98f0 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52bbaf640 10 monclient(hunting): authenticate will time out at 5886.719150s 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52994b640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc5240a98f0 0x7fc5240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33672/0 (socket says 172.21.15.47:33672) 2024-03-25T19:39:31.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.433+0000 7fc52994b640 1 -- 172.21.15.47:0/2105251871 learned_addr learned my addr 172.21.15.47:0/2105251871 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:31.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.453+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533225177 0 0) 0x7fc5240b2a30 con 0x7fc5240a98f0 2024-03-25T19:39:31.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.453+0000 7fc528949640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.453+0000 7fc528949640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.453+0000 7fc528949640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.453+0000 7fc528949640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:39:31.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.453+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc510003520 con 0x7fc5240a98f0 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.472+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435224870 0 0) 0x7fc524142520 con 0x7fc52413f710 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.472+0000 7fc528949640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.472+0000 7fc528949640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.472+0000 7fc528949640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.472+0000 7fc528949640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:39:31.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.472+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5240b2a30 con 0x7fc52413f710 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439595733 0 0) 0x7fc510003520 con 0x7fc5240a98f0 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 >> v1:172.21.15.142:6789/0 conn(0x7fc5241430c0 legacy=0x7fc52414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 >> v1:172.21.15.47:6789/0 conn(0x7fc52413f710 legacy=0x7fc5241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: _finish_hunting 0 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 1 monclient: found mon.noname-b 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240b44c0 con 0x7fc5240a98f0 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: _finish_auth 0 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: _check_auth_tickets 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc520001690 con 0x7fc5240a98f0 2024-03-25T19:39:31.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc52bbaf640 5 monclient: authenticate success, global_id 4427 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc52bbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc528949640 10 monclient: dump: 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:31.486 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:31.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.485+0000 7fc52bbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:31.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5200017f0 con 0x7fc5240a98f0 2024-03-25T19:39:31.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc528949640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:31.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc528949640 1 -- 172.21.15.47:0/2105251871 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc520001c00 con 0x7fc5240a98f0 2024-03-25T19:39:31.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc528949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc528949640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc528949640 10 monclient: dump: 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:31.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 4 set_mon_vals no callback set 2024-03-25T19:39:31.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 10 monclient: shutdown 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 1 -- 172.21.15.47:0/2105251871 >> v1:172.21.15.47:6790/0 conn(0x7fc5240a98f0 legacy=0x7fc5240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 1 -- 172.21.15.47:0/2105251871 shutdown_connections 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.498+0000 7fc52bbaf640 1 -- 172.21.15.47:0/2105251871 >> 172.21.15.47:0/2105251871 conn(0x7fc5240b7ea0 msgr2=0x7fc5240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.499+0000 7fc52bbaf640 1 -- 172.21.15.47:0/2105251871 shutdown_connections 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.499+0000 7fc52bbaf640 1 -- 172.21.15.47:0/2105251871 wait complete. 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.499+0000 7fc52bbaf640 10 monclient: build_initial_monmap 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.499+0000 7fc52bbaf640 10 monclient: monmap: 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:31.500 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 1 librados: starting msgr at 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 1 librados: starting objecter 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 1 Processor -- start 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 1 -- start start 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 1 librados: setting wanted keys 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 1 librados: calling monclient init 2024-03-25T19:39:31.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.500+0000 7fc52bbaf640 10 monclient: init 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient(hunting): picked mon.noname-c con 0x7fc5240a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient(hunting): picked mon.noname-b con 0x7fc52413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient(hunting): picked mon.noname-a con 0x7fc5241430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5241dc070 con 0x7fc5241430c0 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5241395b0 con 0x7fc52413f710 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240ab8b0 con 0x7fc5240a98f0 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.501+0000 7fc52bbaf640 10 monclient(hunting): authenticate will time out at 5886.787164s 2024-03-25T19:39:31.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.509+0000 7fc52914a640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc52413f710 0x7fc5241db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:57186/0 (socket says 172.21.15.47:57186) 2024-03-25T19:39:31.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.509+0000 7fc52914a640 1 -- 172.21.15.47:0/1525093538 learned_addr learned my addr 172.21.15.47:0/1525093538 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:31.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.513+0000 7fc51a7fc640 10 client.?.objecter ms_handle_connect 0x7fc52413f710 2024-03-25T19:39:31.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.513+0000 7fc51a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.519+0000 7fc51a7fc640 10 client.?.objecter ms_handle_connect 0x7fc5240a98f0 2024-03-25T19:39:31.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.519+0000 7fc51a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.521+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026389998 0 0) 0x7fc5241395b0 con 0x7fc52413f710 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.521+0000 7fc51a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.521+0000 7fc51a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.521+0000 7fc51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.521+0000 7fc51a7fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.521+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc508004eb0 con 0x7fc52413f710 2024-03-25T19:39:31.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.530+0000 7fc51a7fc640 10 client.?.objecter ms_handle_connect 0x7fc5241430c0 2024-03-25T19:39:31.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.530+0000 7fc51a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.533+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774115328 0 0) 0x7fc5240ab8b0 con 0x7fc5240a98f0 2024-03-25T19:39:31.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.533+0000 7fc51a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.533+0000 7fc51a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.533+0000 7fc51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.533+0000 7fc51a7fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:39:31.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.533+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5241395b0 con 0x7fc5240a98f0 2024-03-25T19:39:31.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223640369 0 0) 0x7fc508004eb0 con 0x7fc52413f710 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5240ab8b0 con 0x7fc52413f710 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc514001a90 con 0x7fc52413f710 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:31.540 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.540+0000 7fc51a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:31.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.546+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756970376 0 0) 0x7fc5241dc070 con 0x7fc5241430c0 2024-03-25T19:39:31.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.546+0000 7fc51a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.546+0000 7fc51a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.546+0000 7fc51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.546+0000 7fc51a7fc640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:39:31.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.546+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc508004eb0 con 0x7fc5241430c0 2024-03-25T19:39:31.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1374120394 0 0) 0x7fc5240ab8b0 con 0x7fc52413f710 2024-03-25T19:39:31.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 >> v1:172.21.15.47:6790/0 conn(0x7fc5240a98f0 legacy=0x7fc5241375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 >> v1:172.21.15.47:6789/0 conn(0x7fc5241430c0 legacy=0x7fc5241db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 1 monclient: found mon.b 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524138bd0 con 0x7fc52413f710 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc51a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 5 monclient: authenticate success, global_id 4441 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 10 monclient: _renew_subs 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc524138eb0 con 0x7fc52413f710 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 10 monclient: _renew_subs 2024-03-25T19:39:31.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:31.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc524139850 con 0x7fc52413f710 2024-03-25T19:39:31.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.552+0000 7fc52bbaf640 1 librados: init done 2024-03-25T19:39:31.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.553+0000 7fc52bbaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.553+0000 7fc52bbaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.553+0000 7fc52bbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:31.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.553+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5240b3ef0 con 0x7fc52413f710 2024-03-25T19:39:31.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc514001cd0 con 0x7fc52413f710 2024-03-25T19:39:31.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc514003a60 con 0x7fc52413f710 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51a7fc640 10 monclient: dump: 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:31.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.855+0000 7fc51bfff640 4 set_mon_vals no callback set 2024-03-25T19:39:31.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.857+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fc514003c80 con 0x7fc52413f710 2024-03-25T19:39:31.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.857+0000 7fc52994b640 10 monclient: get_auth_request con 0x7fc50806ddb0 auth_method 0 2024-03-25T19:39:31.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.858+0000 7fc51a7fc640 10 client.4441.objecter ms_handle_connect 0x7fc50806ddb0 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.862+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fc5140548e0 con 0x7fc52413f710 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.862+0000 7fc51a7fc640 10 client.4441.objecter ms_dispatch 0x7fc524000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.862+0000 7fc51a7fc640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.862+0000 7fc51a7fc640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.862+0000 7fc51a7fc640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.863+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5140503f0 con 0x7fc52413f710 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.863+0000 7fc51a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:31.863+0000 7fc51a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:32.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.123+0000 7fc52bbaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.123+0000 7fc52bbaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.123+0000 7fc52bbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:32.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.123+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc52413df00 con 0x7fc52413f710 2024-03-25T19:39:32.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.152+0000 7fc51a7fc640 1 -- 172.21.15.47:0/1525093538 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fc514052320 con 0x7fc52413f710 2024-03-25T19:39:32.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.152+0000 7fc51a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.152+0000 7fc51a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:32.152 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:32.152 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:32.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 10 librados: watch_flush enter 2024-03-25T19:39:32.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 10 librados: watch_flush exit 2024-03-25T19:39:32.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T19:39:32.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 10 client.4441.objecter successfully canceled tick 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 >> v1:172.21.15.142:6800/1025302268 conn(0x7fc50806ddb0 legacy=0x7fc508070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 10 monclient: shutdown 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 >> v1:172.21.15.142:6789/0 conn(0x7fc52413f710 legacy=0x7fc5241db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 shutdown_connections 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 >> 172.21.15.47:0/1525093538 conn(0x7fc5240b7ea0 msgr2=0x7fc524074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 shutdown_connections 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 -- 172.21.15.47:0/1525093538 wait complete. 2024-03-25T19:39:32.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:32.154+0000 7fc52bbaf640 1 librados: shutdown 2024-03-25T19:39:32.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:37.943 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:15.179 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:37.951 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:15.187 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:37.988 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:15.224 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.001 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:15.191 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.015 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:15.205 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.026 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:15.261 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.029 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:15.265 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.050 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:15.241 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.067 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:15.303 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.973 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:39.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.071+0000 7ff23710e640 10 monclient: get_monmap_and_config 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.071+0000 7ff23710e640 10 monclient: build_initial_monmap 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.071+0000 7ff23710e640 10 monclient: monmap: 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:39.072372+0000 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:39.072372+0000 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:39.071 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.072+0000 7ff23710e640 1 Processor -- start 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.072+0000 7ff23710e640 1 -- start start 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.072+0000 7ff23710e640 10 monclient: init 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient(hunting): picked mon.noname-b con 0x7ff2300b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2300b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient(hunting): picked mon.noname-c con 0x7ff23014be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300af940 con 0x7ff2300b1730 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300b0a70 con 0x7ff23014be90 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300b0d50 con 0x7ff2300b4d60 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:39.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff23710e640 10 monclient(hunting): authenticate will time out at 5894.359201s 2024-03-25T19:39:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff234eaa640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff2300b4d60 0x7ff23013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51172/0 (socket says 172.21.15.47:51172) 2024-03-25T19:39:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.073+0000 7ff234eaa640 1 -- 172.21.15.47:0/210077240 learned_addr learned my addr 172.21.15.47:0/210077240 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:39.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.096+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705772550 0 0) 0x7ff2300af940 con 0x7ff2300b1730 2024-03-25T19:39:39.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.096+0000 7ff22f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.096+0000 7ff22f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.096+0000 7ff22f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.096+0000 7ff22f7fe640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:39:39.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.096+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff220003520 con 0x7ff2300b1730 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161946194 0 0) 0x7ff220003520 con 0x7ff2300b1730 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 >> v1:172.21.15.47:6790/0 conn(0x7ff2300b4d60 legacy=0x7ff23013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 >> v1:172.21.15.142:6789/0 conn(0x7ff23014be90 legacy=0x7ff23014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 1 monclient: found mon.noname-a 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300b1030 con 0x7ff2300b1730 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:39.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff21c001690 con 0x7ff2300b1730 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff23710e640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff22f7fe640 10 monclient: dump: 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:39.109 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:39.110 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:39.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.108+0000 7ff23710e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff21c001860 con 0x7ff2300b1730 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff22f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff22f7fe640 1 -- 172.21.15.47:0/210077240 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff21c001cb0 con 0x7ff2300b1730 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff23710e640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff23710e640 4 set_mon_vals no callback set 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff22f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff22f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.127+0000 7ff22f7fe640 10 monclient: dump: 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.127 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:39.128 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:39.128 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:39.128 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:39.128 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 10 monclient: shutdown 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 1 -- 172.21.15.47:0/210077240 >> v1:172.21.15.47:6789/0 conn(0x7ff2300b1730 legacy=0x7ff2300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 1 -- 172.21.15.47:0/210077240 shutdown_connections 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 1 -- 172.21.15.47:0/210077240 >> 172.21.15.47:0/210077240 conn(0x7ff2300617a0 msgr2=0x7ff230061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 1 -- 172.21.15.47:0/210077240 shutdown_connections 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.128+0000 7ff23710e640 1 -- 172.21.15.47:0/210077240 wait complete. 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.129+0000 7ff23710e640 10 monclient: build_initial_monmap 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.129+0000 7ff23710e640 10 monclient: monmap: 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.129+0000 7ff23710e640 1 librados: starting msgr at 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.129+0000 7ff23710e640 1 librados: starting objecter 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.130+0000 7ff23710e640 1 Processor -- start 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.130+0000 7ff23710e640 1 -- start start 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.130+0000 7ff23710e640 1 librados: setting wanted keys 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.130+0000 7ff23710e640 1 librados: calling monclient init 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.130+0000 7ff23710e640 10 monclient: init 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient(hunting): picked mon.noname-b con 0x7ff2300b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2300b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient(hunting): picked mon.noname-c con 0x7ff23014be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff230140760 con 0x7ff2300b4d60 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2301435b0 con 0x7ff2300b1730 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff230140380 con 0x7ff23014be90 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff23710e640 10 monclient(hunting): authenticate will time out at 5894.417133s 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff2356ab640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff23014be90 0x7ff2301370c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51216/0 (socket says 172.21.15.47:51216) 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.131+0000 7ff2356ab640 1 -- 172.21.15.47:0/1703328496 learned_addr learned my addr 172.21.15.47:0/1703328496 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.132+0000 7ff22d7fa640 10 client.?.objecter ms_handle_connect 0x7ff23014be90 2024-03-25T19:39:39.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.132+0000 7ff22d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.139+0000 7ff22d7fa640 10 client.?.objecter ms_handle_connect 0x7ff2300b1730 2024-03-25T19:39:39.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.139+0000 7ff22d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.151+0000 7ff22d7fa640 10 client.?.objecter ms_handle_connect 0x7ff2300b4d60 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.151+0000 7ff22d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.171+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814048812 0 0) 0x7ff230140760 con 0x7ff2300b4d60 2024-03-25T19:39:39.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.171+0000 7ff22d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.171+0000 7ff22d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.171+0000 7ff22d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.171+0000 7ff22d7fa640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:39:39.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.171+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff214004eb0 con 0x7ff2300b4d60 2024-03-25T19:39:39.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1855965179 0 0) 0x7ff214004eb0 con 0x7ff2300b4d60 2024-03-25T19:39:39.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff230140760 con 0x7ff2300b4d60 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff21c001b70 con 0x7ff2300b4d60 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 10 monclient(hunting): dump: 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:39.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.233+0000 7ff22d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:39.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.237+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458301882 0 0) 0x7ff2301435b0 con 0x7ff2300b1730 2024-03-25T19:39:39.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.237+0000 7ff22d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.237+0000 7ff22d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.237+0000 7ff22d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.237+0000 7ff22d7fa640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:39:39.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.237+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff214004eb0 con 0x7ff2300b1730 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1343104747 0 0) 0x7ff230140760 con 0x7ff2300b4d60 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 >> v1:172.21.15.47:6790/0 conn(0x7ff23014be90 legacy=0x7ff2301370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 >> v1:172.21.15.142:6789/0 conn(0x7ff2300b1730 legacy=0x7ff2301d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 1 monclient: found mon.a 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff230140510 con 0x7ff2300b4d60 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:39:39.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff22d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 10 monclient: _renew_subs 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2301382b0 con 0x7ff2300b4d60 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 10 monclient: _renew_subs 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2301379b0 con 0x7ff2300b4d60 2024-03-25T19:39:39.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.243+0000 7ff23710e640 1 librados: init done 2024-03-25T19:39:39.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.244+0000 7ff23710e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.244+0000 7ff23710e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.244+0000 7ff23710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:39.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.244+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff230143740 con 0x7ff2300b4d60 2024-03-25T19:39:39.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.255+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff21c001cd0 con 0x7ff2300b4d60 2024-03-25T19:39:39.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.255+0000 7ff22d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:39.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.256+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff21c003b00 con 0x7ff2300b4d60 2024-03-25T19:39:39.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.256+0000 7ff22d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.256+0000 7ff22d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.256+0000 7ff22d7fa640 10 monclient: dump: 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:39.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.256+0000 7ff22effd640 4 set_mon_vals no callback set 2024-03-25T19:39:39.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.577+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ff21c019ad0 con 0x7ff2300b4d60 2024-03-25T19:39:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.577+0000 7ff234eaa640 10 monclient: get_auth_request con 0x7ff21406dce0 auth_method 0 2024-03-25T19:39:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.578+0000 7ff22d7fa640 10 client.4458.objecter ms_handle_connect 0x7ff21406dce0 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7ff21c053760 con 0x7ff2300b4d60 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 10 client.4458.objecter ms_dispatch 0x7ff230000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff21c04f0a0 con 0x7ff2300b4d60 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.583+0000 7ff22d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:39.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.847+0000 7ff23710e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.847+0000 7ff23710e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.847+0000 7ff23710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:39.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.847+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2301401f0 con 0x7ff2300b4d60 2024-03-25T19:39:39.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.869+0000 7ff22d7fa640 1 -- 172.21.15.47:0/1703328496 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7ff21c0511a0 con 0x7ff2300b4d60 2024-03-25T19:39:39.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.869+0000 7ff22d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.869+0000 7ff22d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:39.870 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:39.870 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 10 librados: watch_flush enter 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 10 librados: watch_flush exit 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 10 client.4458.objecter successfully canceled tick 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 >> v1:172.21.15.142:6800/1025302268 conn(0x7ff21406dce0 legacy=0x7ff2140701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 10 monclient: shutdown 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 >> v1:172.21.15.47:6789/0 conn(0x7ff2300b4d60 legacy=0x7ff230142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 shutdown_connections 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 >> 172.21.15.47:0/1703328496 conn(0x7ff2300617a0 msgr2=0x7ff2300b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 shutdown_connections 2024-03-25T19:39:39.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 -- 172.21.15.47:0/1703328496 wait complete. 2024-03-25T19:39:39.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:39.872+0000 7ff23710e640 1 librados: shutdown 2024-03-25T19:39:39.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:46.692 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:46.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.809+0000 7f22a1006640 10 monclient: get_monmap_and_config 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.809+0000 7f22a1006640 10 monclient: build_initial_monmap 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.809+0000 7f22a1006640 10 monclient: monmap: 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:46.811226+0000 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:46.811226+0000 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:46.811 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:46.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 1 Processor -- start 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 1 -- start start 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient: init 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient(hunting): picked mon.noname-c con 0x7f229c0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient(hunting): picked mon.noname-b con 0x7f229c0b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient(hunting): picked mon.noname-a con 0x7f229c14be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0af940 con 0x7f229c14be90 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0b0a70 con 0x7f229c0b4d60 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0b0d50 con 0x7f229c0b1730 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:46.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.810+0000 7f22a1006640 10 monclient(hunting): authenticate will time out at 5902.097843s 2024-03-25T19:39:46.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.818+0000 7f229b59e640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f229c14be90 0x7f229c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50362/0 (socket says 172.21.15.47:50362) 2024-03-25T19:39:46.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.818+0000 7f229b59e640 1 -- 172.21.15.47:0/4215735830 learned_addr learned my addr 172.21.15.47:0/4215735830 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:46.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.828+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267274792 0 0) 0x7f229c0af940 con 0x7f229c14be90 2024-03-25T19:39:46.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.828+0000 7f2299d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.828+0000 7f2299d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.828+0000 7f2299d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.828+0000 7f2299d9b640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:39:46.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.828+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2284003520 con 0x7f229c14be90 2024-03-25T19:39:46.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.830+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908223514 0 0) 0x7f229c0b0d50 con 0x7f229c0b1730 2024-03-25T19:39:46.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.830+0000 7f2299d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.830+0000 7f2299d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.830+0000 7f2299d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.830+0000 7f2299d9b640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:39:46.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.830+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f229c0af940 con 0x7f229c0b1730 2024-03-25T19:39:46.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.833+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395522378 0 0) 0x7f229c0b0a70 con 0x7f229c0b4d60 2024-03-25T19:39:46.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.833+0000 7f2299d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.833+0000 7f2299d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.833+0000 7f2299d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.833+0000 7f2299d9b640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:39:46.834 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.833+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22840039d0 con 0x7f229c0b4d60 2024-03-25T19:39:46.838 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162188208 0 0) 0x7f2284003520 con 0x7f229c14be90 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 >> v1:172.21.15.47:6790/0 conn(0x7f229c0b1730 legacy=0x7f229c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 >> v1:172.21.15.142:6789/0 conn(0x7f229c0b4d60 legacy=0x7f229c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 1 monclient: found mon.noname-a 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c0b1030 con 0x7f229c14be90 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: _finish_auth 0 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2294001a10 con 0x7f229c14be90 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f22a1006640 5 monclient: authenticate success, global_id 4461 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f22a1006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f2299d9b640 10 monclient: dump: 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.838+0000 7f22a1006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2294001b70 con 0x7f229c14be90 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f2299d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f2299d9b640 1 -- 172.21.15.47:0/4215735830 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2294001f80 con 0x7f229c14be90 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f2299d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f2299d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f2299d9b640 10 monclient: dump: 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f22a1006640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:46.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.845+0000 7f22a1006640 4 set_mon_vals no callback set 2024-03-25T19:39:46.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 10 monclient: shutdown 2024-03-25T19:39:46.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:46.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 1 -- 172.21.15.47:0/4215735830 >> v1:172.21.15.47:6789/0 conn(0x7f229c14be90 legacy=0x7f229c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 1 -- 172.21.15.47:0/4215735830 shutdown_connections 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 1 -- 172.21.15.47:0/4215735830 >> 172.21.15.47:0/4215735830 conn(0x7f229c0617a0 msgr2=0x7f229c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 1 -- 172.21.15.47:0/4215735830 shutdown_connections 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.846+0000 7f22a1006640 1 -- 172.21.15.47:0/4215735830 wait complete. 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 10 monclient: build_initial_monmap 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 10 monclient: monmap: 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:46.850 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 1 librados: starting msgr at 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 1 librados: starting objecter 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 1 Processor -- start 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 1 -- start start 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 1 librados: setting wanted keys 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 1 librados: calling monclient init 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.847+0000 7f22a1006640 10 monclient: init 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient(hunting): picked mon.noname-a con 0x7f229c0b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient(hunting): picked mon.noname-b con 0x7f229c0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient(hunting): picked mon.noname-c con 0x7f229c14be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c140520 con 0x7f229c0b1730 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c140a40 con 0x7f229c0b4d60 2024-03-25T19:39:46.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c136fd0 con 0x7f229c14be90 2024-03-25T19:39:46.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:46.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.848+0000 7f22a1006640 10 monclient(hunting): authenticate will time out at 5902.135293s 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.853+0000 7f229ad9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f229c0b1730 0x7f229c135b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50400/0 (socket says 172.21.15.47:50400) 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.854+0000 7f229ad9d640 1 -- 172.21.15.47:0/1824244112 learned_addr learned my addr 172.21.15.47:0/1824244112 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:46.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.858+0000 7f22837fe640 10 client.?.objecter ms_handle_connect 0x7f229c0b1730 2024-03-25T19:39:46.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.858+0000 7f22837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.861+0000 7f22837fe640 10 client.?.objecter ms_handle_connect 0x7f229c0b4d60 2024-03-25T19:39:46.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.861+0000 7f22837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.869+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885798551 0 0) 0x7f229c140520 con 0x7f229c0b1730 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.869+0000 7f22837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.869+0000 7f22837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.869+0000 7f22837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.869+0000 7f22837fe640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.869+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288004eb0 con 0x7f229c0b1730 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.870+0000 7f22837fe640 10 client.?.objecter ms_handle_connect 0x7f229c14be90 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.870+0000 7f22837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.878+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742068031 0 0) 0x7f2288004eb0 con 0x7f229c0b1730 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.878+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f229c140520 con 0x7f229c0b1730 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.878+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f228c001a30 con 0x7f229c0b1730 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.878+0000 7f22837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.879+0000 7f22837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.879+0000 7f22837fe640 10 monclient(hunting): dump: 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:46.879 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:46.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.879+0000 7f22837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:46.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.880+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165954904 0 0) 0x7f229c140a40 con 0x7f229c0b4d60 2024-03-25T19:39:46.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.880+0000 7f22837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.880+0000 7f22837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.880+0000 7f22837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.880+0000 7f22837fe640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:39:46.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.880+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288004eb0 con 0x7f229c0b4d60 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.890+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3054568833 0 0) 0x7f229c140520 con 0x7f229c0b1730 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.890+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 >> v1:172.21.15.47:6790/0 conn(0x7f229c14be90 legacy=0x7f229c1e2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.890+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 >> v1:172.21.15.142:6789/0 conn(0x7f229c0b4d60 legacy=0x7f229c134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 1 monclient: found mon.a 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c137160 con 0x7f229c0b1730 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:46.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 10 monclient: _renew_subs 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c136210 con 0x7f229c0b1730 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 10 monclient: _renew_subs 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c135730 con 0x7f229c0b1730 2024-03-25T19:39:46.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.891+0000 7f22a1006640 1 librados: init done 2024-03-25T19:39:46.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.892+0000 7f22a1006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:46.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.892+0000 7f22a1006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:46.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.892+0000 7f22a1006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:46.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:46.892+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c1406b0 con 0x7f229c0b1730 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f228c001cb0 con 0x7f229c0b1730 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f22837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f228c003a60 con 0x7f229c0b1730 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f22837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f22837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f22837fe640 10 monclient: dump: 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:47.217 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.216+0000 7f229959a640 4 set_mon_vals no callback set 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.217+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f228c003c40 con 0x7f229c0b1730 2024-03-25T19:39:47.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.217+0000 7f229a59c640 10 monclient: get_auth_request con 0x7f228806de80 auth_method 0 2024-03-25T19:39:47.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.218+0000 7f22837fe640 10 client.4464.objecter ms_handle_connect 0x7f228806de80 2024-03-25T19:39:47.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f228c0548a0 con 0x7f229c0b1730 2024-03-25T19:39:47.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 10 client.4464.objecter ms_dispatch 0x7f229c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:47.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:47.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:47.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-25T19:39:47.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f228c0503b0 con 0x7f229c0b1730 2024-03-25T19:39:47.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:47.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.223+0000 7f22837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:47.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.478+0000 7f22a1006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.479+0000 7f22a1006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.479+0000 7f22a1006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:39:47.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.479+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f229c0627b0 con 0x7f229c0b1730 2024-03-25T19:39:47.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.500+0000 7f22837fe640 1 -- 172.21.15.47:0/1824244112 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f228c0522e0 con 0x7f229c0b1730 2024-03-25T19:39:47.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.500+0000 7f22837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.500+0000 7f22837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:47.501 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:47.501 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:47.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 10 librados: watch_flush enter 2024-03-25T19:39:47.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 10 librados: watch_flush exit 2024-03-25T19:39:47.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-25T19:39:47.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 10 client.4464.objecter successfully canceled tick 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 >> v1:172.21.15.142:6800/1025302268 conn(0x7f228806de80 legacy=0x7f2288070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 10 monclient: shutdown 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 >> v1:172.21.15.47:6789/0 conn(0x7f229c0b1730 legacy=0x7f229c135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 shutdown_connections 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 >> 172.21.15.47:0/1824244112 conn(0x7f229c0617a0 msgr2=0x7f229c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 shutdown_connections 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 -- 172.21.15.47:0/1824244112 wait complete. 2024-03-25T19:39:47.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:47.502+0000 7f22a1006640 1 librados: shutdown 2024-03-25T19:39:47.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:54.320 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:54.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.438+0000 7f8ab69f6640 10 monclient: get_monmap_and_config 2024-03-25T19:39:54.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.438+0000 7f8ab69f6640 10 monclient: build_initial_monmap 2024-03-25T19:39:54.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.438+0000 7f8ab69f6640 10 monclient: monmap: 2024-03-25T19:39:54.438 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:54.438 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.438 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:39:54.439452+0000 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:39:54.439452+0000 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:39:54.439 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.439+0000 7f8ab69f6640 1 Processor -- start 2024-03-25T19:39:54.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.439+0000 7f8ab69f6640 1 -- start start 2024-03-25T19:39:54.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.439+0000 7f8ab69f6640 10 monclient: init 2024-03-25T19:39:54.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:54.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:54.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab00a96e0 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab00b1b10 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab0146060 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab0140040 con 0x7f8ab00b1b10 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab00b1fa0 con 0x7f8ab0146060 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab00b24c0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:54.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.440+0000 7f8ab69f6640 10 monclient(hunting): authenticate will time out at 5909.726513s 2024-03-25T19:39:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.445+0000 7f8ab4f93640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8ab0146060 0x7f8ab014dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:49924/0 (socket says 172.21.15.47:49924) 2024-03-25T19:39:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.445+0000 7f8ab4f93640 1 -- 172.21.15.47:0/16171669 learned_addr learned my addr 172.21.15.47:0/16171669 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:54.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.517+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726680893 0 0) 0x7f8ab0140040 con 0x7f8ab00b1b10 2024-03-25T19:39:54.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.517+0000 7f8aaeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.517+0000 7f8aaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.517+0000 7f8aaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.517+0000 7f8aaeffd640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:39:54.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.517+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a98003520 con 0x7f8ab00b1b10 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 346390324 0 0) 0x7f8a98003520 con 0x7f8ab00b1b10 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 >> v1:172.21.15.47:6790/0 conn(0x7f8ab00a96e0 legacy=0x7f8ab00b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 >> v1:172.21.15.142:6789/0 conn(0x7f8ab0146060 legacy=0x7f8ab014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 1 monclient: found mon.noname-a 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab00b15b0 con 0x7f8ab00b1b10 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aa4001700 con 0x7f8ab00b1b10 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8aaeffd640 10 monclient: dump: 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8ab69f6640 5 monclient: authenticate success, global_id 4467 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.526+0000 7f8ab69f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:54.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aa4001860 con 0x7f8ab00b1b10 2024-03-25T19:39:54.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8aaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8aaeffd640 1 -- 172.21.15.47:0/16171669 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aa4001cb0 con 0x7f8ab00b1b10 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8ab69f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8ab69f6640 4 set_mon_vals no callback set 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8aaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8aaeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.545+0000 7f8aaeffd640 10 monclient: dump: 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.546 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 10 monclient: shutdown 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 1 -- 172.21.15.47:0/16171669 >> v1:172.21.15.47:6789/0 conn(0x7f8ab00b1b10 legacy=0x7f8ab0145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 1 -- 172.21.15.47:0/16171669 shutdown_connections 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 1 -- 172.21.15.47:0/16171669 >> 172.21.15.47:0/16171669 conn(0x7f8ab00b7ea0 msgr2=0x7f8ab00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 1 -- 172.21.15.47:0/16171669 shutdown_connections 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.546+0000 7f8ab69f6640 1 -- 172.21.15.47:0/16171669 wait complete. 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 10 monclient: build_initial_monmap 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 10 monclient: monmap: 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:54.549 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 1 librados: starting msgr at 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 1 librados: starting objecter 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 1 Processor -- start 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 1 -- start start 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 1 librados: setting wanted keys 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 1 librados: calling monclient init 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.547+0000 7f8ab69f6640 10 monclient: init 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab00a96e0 addr v1:172.21.15.142:6789/0 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab0146060 addr v1:172.21.15.47:6789/0 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab0142370 addr v1:172.21.15.47:6790/0 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab0143560 con 0x7f8ab0146060 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab013d920 con 0x7f8ab00a96e0 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab013dab0 con 0x7f8ab0142370 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.548+0000 7f8ab69f6640 10 monclient(hunting): authenticate will time out at 5909.834198s 2024-03-25T19:39:54.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.563+0000 7f8aaf7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8ab0146060 0x7f8ab01da9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37770/0 (socket says 172.21.15.47:37770) 2024-03-25T19:39:54.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.563+0000 7f8aaf7fe640 1 -- 172.21.15.47:0/3551864490 learned_addr learned my addr 172.21.15.47:0/3551864490 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:39:54.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.564+0000 7f8aacff9640 10 client.?.objecter ms_handle_connect 0x7f8ab0146060 2024-03-25T19:39:54.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.564+0000 7f8aacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.577+0000 7f8aacff9640 10 client.?.objecter ms_handle_connect 0x7f8ab0142370 2024-03-25T19:39:54.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.577+0000 7f8aacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 10 client.?.objecter ms_handle_connect 0x7f8ab00a96e0 2024-03-25T19:39:54.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233553445 0 0) 0x7f8ab0143560 con 0x7f8ab0146060 2024-03-25T19:39:54.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:39:54.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.584+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c004eb0 con 0x7f8ab0146060 2024-03-25T19:39:54.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.590+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637086690 0 0) 0x7f8ab013d920 con 0x7f8ab00a96e0 2024-03-25T19:39:54.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.590+0000 7f8aacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.590+0000 7f8aacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.590+0000 7f8aacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.590+0000 7f8aacff9640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:39:54.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.590+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab0143560 con 0x7f8ab00a96e0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1219533962 0 0) 0x7f8ab0143560 con 0x7f8ab00a96e0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ab013d920 con 0x7f8ab00a96e0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aa00018b0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 10 monclient(hunting): dump: 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.596+0000 7f8aacff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.597+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347705853 0 0) 0x7f8ab013dab0 con 0x7f8ab0142370 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.597+0000 7f8aacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.597+0000 7f8aacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.597+0000 7f8aacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.597+0000 7f8aacff9640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:39:54.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.597+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab0143560 con 0x7f8ab0142370 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.609+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65042310 0 0) 0x7f8ab0143560 con 0x7f8ab0142370 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.609+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ab013dab0 con 0x7f8ab0142370 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.609+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aa8001a60 con 0x7f8ab0142370 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.609+0000 7f8aacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.609+0000 7f8aacff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.609+0000 7f8aacff9640 10 monclient(hunting): dump: 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.609 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.610 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.610 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.610 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:54.610 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:54.610 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:54.610 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.613+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051710500 0 0) 0x7f8a9c004eb0 con 0x7f8ab0146060 2024-03-25T19:39:54.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.613+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ab0143560 con 0x7f8ab0146060 2024-03-25T19:39:54.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.613+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aa4001c70 con 0x7f8ab0146060 2024-03-25T19:39:54.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.613+0000 7f8aacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.613+0000 7f8aacff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.613+0000 7f8aacff9640 10 monclient(hunting): dump: 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:54.613 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2579908499 0 0) 0x7f8ab013d920 con 0x7f8ab00a96e0 2024-03-25T19:39:54.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 >> v1:172.21.15.47:6790/0 conn(0x7f8ab0142370 legacy=0x7f8ab01e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 >> v1:172.21.15.47:6789/0 conn(0x7f8ab0146060 legacy=0x7f8ab01da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 1 monclient: found mon.b 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab013dc40 con 0x7f8ab00a96e0 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8aacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 10 monclient: _renew_subs 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab01386f0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 10 monclient: _renew_subs 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab01db0d0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.623+0000 7f8ab69f6640 1 librados: init done 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.624+0000 7f8ab69f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.624+0000 7f8ab69f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.624+0000 7f8ab69f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:54.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.624+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab00a9ac0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.840+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aa0003610 con 0x7f8ab00a96e0 2024-03-25T19:39:54.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.840+0000 7f8aacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:54.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.840+0000 7f8aae7fc640 4 set_mon_vals no callback set 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.909+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8aa0003a60 con 0x7f8ab00a96e0 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.909+0000 7f8aacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.909+0000 7f8aacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.909+0000 7f8aacff9640 10 monclient: dump: 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:39:54.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.909+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f8aa0003c40 con 0x7f8ab00a96e0 2024-03-25T19:39:54.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.910+0000 7f8aaf7fe640 10 monclient: get_auth_request con 0x7f8a9c06e110 auth_method 0 2024-03-25T19:39:54.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.910+0000 7f8aacff9640 10 client.4453.objecter ms_handle_connect 0x7f8a9c06e110 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.914+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f8aa00535b0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.914+0000 7f8aacff9640 10 client.4453.objecter ms_dispatch 0x7f8ab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.914+0000 7f8aacff9640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.914+0000 7f8aacff9640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.914+0000 7f8aacff9640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T19:39:54.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.920+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8aa004f0c0 con 0x7f8ab00a96e0 2024-03-25T19:39:54.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.920+0000 7f8aacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:54.920+0000 7f8aacff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:55.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.181+0000 7f8ab69f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:55.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.181+0000 7f8ab69f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:55.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.181+0000 7f8ab69f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:39:55.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.181+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ab00627b0 con 0x7f8ab00a96e0 2024-03-25T19:39:55.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.199+0000 7f8aacff9640 1 -- 172.21.15.47:0/3551864490 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f8aa0050ff0 con 0x7f8ab00a96e0 2024-03-25T19:39:55.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.199+0000 7f8aacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:55.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.199+0000 7f8aacff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:55.200 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:39:55.200 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:55.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 10 librados: watch_flush enter 2024-03-25T19:39:55.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 10 librados: watch_flush exit 2024-03-25T19:39:55.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T19:39:55.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 10 client.4453.objecter successfully canceled tick 2024-03-25T19:39:55.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 >> v1:172.21.15.142:6800/1025302268 conn(0x7f8a9c06e110 legacy=0x7f8a9c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 10 monclient: shutdown 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 >> v1:172.21.15.142:6789/0 conn(0x7f8ab00a96e0 legacy=0x7f8ab0137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 shutdown_connections 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 >> 172.21.15.47:0/3551864490 conn(0x7f8ab00b7ea0 msgr2=0x7f8ab013e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 shutdown_connections 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 -- 172.21.15.47:0/3551864490 wait complete. 2024-03-25T19:39:55.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:39:55.202+0000 7f8ab69f6640 1 librados: shutdown 2024-03-25T19:39:55.214 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:57.959 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:35.195 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.017 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:35.252 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.062 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:35.297 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.072 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:35.262 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.076 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:35.311 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.084 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:35.319 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.086 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:35.276 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.108 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:35.344 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.116 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:35.306 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:02.022 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:02.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.135+0000 7f2907e86640 10 monclient: get_monmap_and_config 2024-03-25T19:40:02.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.135+0000 7f2907e86640 10 monclient: build_initial_monmap 2024-03-25T19:40:02.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.135+0000 7f2907e86640 10 monclient: monmap: 2024-03-25T19:40:02.136 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:02.136 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:02.136 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:02.136665+0000 2024-03-25T19:40:02.136 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:02.136665+0000 2024-03-25T19:40:02.136 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:02.136 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:02.137 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:02.137 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:02.137 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:02.137 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:02.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.136+0000 7f2907e86640 1 Processor -- start 2024-03-25T19:40:02.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.136+0000 7f2907e86640 1 -- start start 2024-03-25T19:40:02.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.136+0000 7f2907e86640 10 monclient: init 2024-03-25T19:40:02.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:02.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:02.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient(hunting): picked mon.noname-b con 0x7f29000a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient(hunting): picked mon.noname-c con 0x7f290013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient(hunting): picked mon.noname-a con 0x7f29001430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2900142520 con 0x7f29001430c0 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000b28a0 con 0x7f290013f710 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000b2a30 con 0x7f29000a98f0 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:02.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.137+0000 7f2907e86640 10 monclient(hunting): authenticate will time out at 5917.423423s 2024-03-25T19:40:02.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.149+0000 7f2905421640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f290013f710 0x7f29001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:39260/0 (socket says 172.21.15.47:39260) 2024-03-25T19:40:02.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.149+0000 7f2905421640 1 -- 172.21.15.47:0/3833692483 learned_addr learned my addr 172.21.15.47:0/3833692483 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:02.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.159+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589470732 0 0) 0x7f29000b28a0 con 0x7f290013f710 2024-03-25T19:40:02.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.159+0000 7f2904c20640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.159+0000 7f2904c20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.159+0000 7f2904c20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.159+0000 7f2904c20640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:40:02.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.159+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec003520 con 0x7f290013f710 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1808996137 0 0) 0x7f28ec003520 con 0x7f290013f710 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 >> v1:172.21.15.47:6790/0 conn(0x7f29000a98f0 legacy=0x7f29000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 >> v1:172.21.15.47:6789/0 conn(0x7f29001430c0 legacy=0x7f290014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: _finish_hunting 0 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 1 monclient: found mon.noname-c 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000b44c0 con 0x7f290013f710 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: _finish_auth 0 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: _check_auth_tickets 2024-03-25T19:40:02.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2907e86640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f28f0001690 con 0x7f290013f710 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2907e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2904c20640 10 monclient: dump: 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:02.182 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:02.183 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:02.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.181+0000 7f2907e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:02.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.194+0000 7f2904c20640 1 -- 172.21.15.47:0/3833692483 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28f00017f0 con 0x7f290013f710 2024-03-25T19:40:02.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.194+0000 7f2904c20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:02.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.194+0000 7f2907e86640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:02.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.194+0000 7f2907e86640 4 set_mon_vals no callback set 2024-03-25T19:40:02.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 10 monclient: shutdown 2024-03-25T19:40:02.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:02.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 1 -- 172.21.15.47:0/3833692483 >> v1:172.21.15.142:6789/0 conn(0x7f290013f710 legacy=0x7f29001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 1 -- 172.21.15.47:0/3833692483 shutdown_connections 2024-03-25T19:40:02.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 1 -- 172.21.15.47:0/3833692483 >> 172.21.15.47:0/3833692483 conn(0x7f29000b7ea0 msgr2=0x7f29000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:02.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 1 -- 172.21.15.47:0/3833692483 shutdown_connections 2024-03-25T19:40:02.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.195+0000 7f2907e86640 1 -- 172.21.15.47:0/3833692483 wait complete. 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.196+0000 7f2907e86640 10 monclient: build_initial_monmap 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.196+0000 7f2907e86640 10 monclient: monmap: 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:02.196 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.196+0000 7f2907e86640 1 librados: starting msgr at 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.196+0000 7f2907e86640 1 librados: starting objecter 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.196+0000 7f2907e86640 1 Processor -- start 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 1 -- start start 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 1 librados: setting wanted keys 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 1 librados: calling monclient init 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 10 monclient: init 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 10 monclient(hunting): picked mon.noname-a con 0x7f29000a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.197+0000 7f2907e86640 10 monclient(hunting): picked mon.noname-b con 0x7f290013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.198+0000 7f2907e86640 10 monclient(hunting): picked mon.noname-c con 0x7f29001430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.198+0000 7f2907e86640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2900137140 con 0x7f29000a98f0 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.198+0000 7f2907e86640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29001d7b90 con 0x7f290013f710 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.198+0000 7f2907e86640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29001d7590 con 0x7f29001430c0 2024-03-25T19:40:02.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.198+0000 7f2907e86640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:02.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.198+0000 7f2907e86640 10 monclient(hunting): authenticate will time out at 5917.484078s 2024-03-25T19:40:02.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.201+0000 7f2905c22640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f29000a98f0 0x7f29001394b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60736/0 (socket says 172.21.15.47:60736) 2024-03-25T19:40:02.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.201+0000 7f2905c22640 1 -- 172.21.15.47:0/1603371596 learned_addr learned my addr 172.21.15.47:0/1603371596 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:02.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.202+0000 7f28f67fc640 10 client.?.objecter ms_handle_connect 0x7f29000a98f0 2024-03-25T19:40:02.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.202+0000 7f28f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:02.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.209+0000 7f28f67fc640 10 client.?.objecter ms_handle_connect 0x7f290013f710 2024-03-25T19:40:02.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.209+0000 7f28f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:02.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.210+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100529227 0 0) 0x7f2900137140 con 0x7f29000a98f0 2024-03-25T19:40:02.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.210+0000 7f28f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.210+0000 7f28f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.210+0000 7f28f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.210+0000 7f28f67fc640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:40:02.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.210+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e4004eb0 con 0x7f29000a98f0 2024-03-25T19:40:02.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.216+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305129008 0 0) 0x7f29001d7b90 con 0x7f290013f710 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.216+0000 7f28f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.216+0000 7f28f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.216+0000 7f28f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.216+0000 7f28f67fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.216+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2900137140 con 0x7f290013f710 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 259434554 0 0) 0x7f28e4004eb0 con 0x7f29000a98f0 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29001d7b90 con 0x7f29000a98f0 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f28fc0019d0 con 0x7f29000a98f0 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 10 monclient(hunting): dump: 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.218+0000 7f28f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:02.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.225+0000 7f28f67fc640 10 client.?.objecter ms_handle_connect 0x7f29001430c0 2024-03-25T19:40:02.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.225+0000 7f28f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:02.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703650744 0 0) 0x7f29001d7b90 con 0x7f29000a98f0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 >> v1:172.21.15.47:6790/0 conn(0x7f29001430c0 legacy=0x7f29001e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 >> v1:172.21.15.142:6789/0 conn(0x7f290013f710 legacy=0x7f2900136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 1 monclient: found mon.a 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29001d7720 con 0x7f29000a98f0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f28f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 10 monclient: _renew_subs 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290013b7d0 con 0x7f29000a98f0 2024-03-25T19:40:02.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 10 monclient: _renew_subs 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29001d8290 con 0x7f29000a98f0 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.228+0000 7f2907e86640 1 librados: init done 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.229+0000 7f2907e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.229+0000 7f2907e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.229+0000 7f2907e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:02.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.229+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290013bb70 con 0x7f29000a98f0 2024-03-25T19:40:02.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28fc0035f0 con 0x7f29000a98f0 2024-03-25T19:40:02.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:02.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f28fc003a00 con 0x7f29000a98f0 2024-03-25T19:40:02.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:02.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f67fc640 10 monclient: dump: 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:02.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.549+0000 7f28f7fff640 4 set_mon_vals no callback set 2024-03-25T19:40:02.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.550+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f28fc003c20 con 0x7f29000a98f0 2024-03-25T19:40:02.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.551+0000 7f2905421640 10 monclient: get_auth_request con 0x7f28e406dd40 auth_method 0 2024-03-25T19:40:02.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.551+0000 7f28f67fc640 10 client.4476.objecter ms_handle_connect 0x7f28e406dd40 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f28fc0547d0 con 0x7f29000a98f0 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 10 client.4476.objecter ms_dispatch 0x7f2900000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28fc0502e0 con 0x7f29000a98f0 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.557+0000 7f28f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.776+0000 7f2907e86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.776+0000 7f2907e86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.776+0000 7f2907e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.776+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29001d7d20 con 0x7f29000a98f0 2024-03-25T19:40:02.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.866+0000 7f28f67fc640 1 -- 172.21.15.47:0/1603371596 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f28fc052210 con 0x7f29000a98f0 2024-03-25T19:40:02.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.866+0000 7f28f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.866+0000 7f28f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 10 librados: watch_flush enter 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 10 librados: watch_flush exit 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 10 client.4476.objecter successfully canceled tick 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 >> v1:172.21.15.142:6800/1025302268 conn(0x7f28e406dd40 legacy=0x7f28e4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 10 monclient: shutdown 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 >> v1:172.21.15.47:6789/0 conn(0x7f29000a98f0 legacy=0x7f29001394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 shutdown_connections 2024-03-25T19:40:02.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 >> 172.21.15.47:0/1603371596 conn(0x7f29000b7ea0 msgr2=0x7f2900074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:02.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 shutdown_connections 2024-03-25T19:40:02.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 -- 172.21.15.47:0/1603371596 wait complete. 2024-03-25T19:40:02.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:02.868+0000 7f2907e86640 1 librados: shutdown 2024-03-25T19:40:02.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:09.687 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.797+0000 7f539a93b640 10 monclient: get_monmap_and_config 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.797+0000 7f539a93b640 10 monclient: build_initial_monmap 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.797+0000 7f539a93b640 10 monclient: monmap: 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:09.798606+0000 2024-03-25T19:40:09.797 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:09.798606+0000 2024-03-25T19:40:09.798 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:09.798 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:09.798 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:09.798 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:09.798 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:09.798 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.798+0000 7f539a93b640 1 Processor -- start 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.798+0000 7f539a93b640 1 -- start start 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.798+0000 7f539a93b640 10 monclient: init 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 10 monclient(hunting): picked mon.noname-c con 0x7f53940a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:09.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 10 monclient(hunting): picked mon.noname-b con 0x7f539413f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:09.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 10 monclient(hunting): picked mon.noname-a con 0x7f53941430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:09.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5394142520 con 0x7f53941430c0 2024-03-25T19:40:09.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.799+0000 7f539a93b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940b28a0 con 0x7f53940a98f0 2024-03-25T19:40:09.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.800+0000 7f539a93b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940b2a30 con 0x7f539413f710 2024-03-25T19:40:09.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.800+0000 7f539a93b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:09.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.800+0000 7f539a93b640 10 monclient(hunting): authenticate will time out at 5925.086114s 2024-03-25T19:40:09.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.807+0000 7f5393fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f53940a98f0 0x7f53940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:59878/0 (socket says 172.21.15.47:59878) 2024-03-25T19:40:09.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.807+0000 7f5393fff640 1 -- 172.21.15.47:0/508594859 learned_addr learned my addr 172.21.15.47:0/508594859 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:09.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.821+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264122152 0 0) 0x7f5394142520 con 0x7f53941430c0 2024-03-25T19:40:09.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.821+0000 7f5392ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.821+0000 7f5392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.821+0000 7f5392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.821+0000 7f5392ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.821+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c003520 con 0x7f53941430c0 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.822+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495144974 0 0) 0x7f53940b28a0 con 0x7f53940a98f0 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.822+0000 7f5392ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.822+0000 7f5392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.822+0000 7f5392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.822+0000 7f5392ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:40:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.822+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5394142520 con 0x7f53940a98f0 2024-03-25T19:40:09.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.828+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053210398 0 0) 0x7f53940b2a30 con 0x7f539413f710 2024-03-25T19:40:09.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.828+0000 7f5392ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.828+0000 7f5392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.828+0000 7f5392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.828+0000 7f5392ffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:40:09.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.828+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53940b28a0 con 0x7f539413f710 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1810876014 0 0) 0x7f537c003520 con 0x7f53941430c0 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 >> v1:172.21.15.47:6790/0 conn(0x7f539413f710 legacy=0x7f53941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 >> v1:172.21.15.142:6789/0 conn(0x7f53940a98f0 legacy=0x7f53940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 1 monclient: found mon.noname-a 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940b44c0 con 0x7f53941430c0 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:09.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f538c001a10 con 0x7f53941430c0 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f539a93b640 5 monclient: authenticate success, global_id 4479 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f5392ffd640 10 monclient: dump: 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:09.833 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:09.834 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:09.834 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.832+0000 7f539a93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f538c001b70 con 0x7f53941430c0 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f5392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f5392ffd640 1 -- 172.21.15.47:0/508594859 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f538c001f80 con 0x7f53941430c0 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f5392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f5392ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f5392ffd640 10 monclient: dump: 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.847 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:09.848 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:09.848 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:09.848 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:09.848 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:09.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f539a93b640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:09.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.847+0000 7f539a93b640 4 set_mon_vals no callback set 2024-03-25T19:40:09.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 10 monclient: shutdown 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 1 -- 172.21.15.47:0/508594859 >> v1:172.21.15.47:6789/0 conn(0x7f53941430c0 legacy=0x7f539414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 1 -- 172.21.15.47:0/508594859 shutdown_connections 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 1 -- 172.21.15.47:0/508594859 >> 172.21.15.47:0/508594859 conn(0x7f53940b7ea0 msgr2=0x7f53940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 1 -- 172.21.15.47:0/508594859 shutdown_connections 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.848+0000 7f539a93b640 1 -- 172.21.15.47:0/508594859 wait complete. 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.849+0000 7f539a93b640 10 monclient: build_initial_monmap 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.849+0000 7f539a93b640 10 monclient: monmap: 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:09.850 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 1 librados: starting msgr at 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 1 librados: starting objecter 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 1 Processor -- start 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 1 -- start start 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 1 librados: setting wanted keys 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 1 librados: calling monclient init 2024-03-25T19:40:09.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.850+0000 7f539a93b640 10 monclient: init 2024-03-25T19:40:09.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:09.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient(hunting): picked mon.noname-c con 0x7f53940a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient(hunting): picked mon.noname-b con 0x7f539413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient(hunting): picked mon.noname-a con 0x7f53941430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539413b410 con 0x7f53941430c0 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539413b010 con 0x7f539413f710 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5394135230 con 0x7f53940a98f0 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:09.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.851+0000 7f539a93b640 10 monclient(hunting): authenticate will time out at 5925.137836s 2024-03-25T19:40:09.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.859+0000 7f5393fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f53940a98f0 0x7f5394130ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45268/0 (socket says 172.21.15.47:45268) 2024-03-25T19:40:09.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.859+0000 7f5393fff640 1 -- 172.21.15.47:0/26189090 learned_addr learned my addr 172.21.15.47:0/26189090 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:09.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.861+0000 7f5390ff9640 10 client.?.objecter ms_handle_connect 0x7f53940a98f0 2024-03-25T19:40:09.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.861+0000 7f5390ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.867+0000 7f5390ff9640 10 client.?.objecter ms_handle_connect 0x7f539413f710 2024-03-25T19:40:09.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.867+0000 7f5390ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.875+0000 7f5390ff9640 10 client.?.objecter ms_handle_connect 0x7f53941430c0 2024-03-25T19:40:09.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.875+0000 7f5390ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.876+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132078919 0 0) 0x7f539413b010 con 0x7f539413f710 2024-03-25T19:40:09.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.876+0000 7f5390ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.876+0000 7f5390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.876+0000 7f5390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.876+0000 7f5390ff9640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:40:09.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.876+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5380004eb0 con 0x7f539413f710 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.879+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859084824 0 0) 0x7f5394135230 con 0x7f53940a98f0 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.879+0000 7f5390ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.879+0000 7f5390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.879+0000 7f5390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.879+0000 7f5390ff9640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.879+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f539413b010 con 0x7f53940a98f0 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525661218 0 0) 0x7f5380004eb0 con 0x7f539413f710 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5394135230 con 0x7f539413f710 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f53880018f0 con 0x7f539413f710 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:09.884 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:09.885 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:09.885 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:09.885 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:09.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.884+0000 7f5390ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:09.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.889+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26799064 0 0) 0x7f539413b410 con 0x7f53941430c0 2024-03-25T19:40:09.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.889+0000 7f5390ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.889+0000 7f5390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.889+0000 7f5390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.889+0000 7f5390ff9640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:40:09.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.889+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5380004eb0 con 0x7f53941430c0 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.896+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1772143741 0 0) 0x7f539413b010 con 0x7f53940a98f0 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f539413b410 con 0x7f53940a98f0 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5384001970 con 0x7f53940a98f0 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:09.897 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 707548358 0 0) 0x7f5394135230 con 0x7f539413f710 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 >> v1:172.21.15.47:6790/0 conn(0x7f53940a98f0 legacy=0x7f5394130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 >> v1:172.21.15.47:6789/0 conn(0x7f53941430c0 legacy=0x7f53941db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 monclient: found mon.b 2024-03-25T19:40:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53941353c0 con 0x7f539413f710 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f5390ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 10 monclient: _renew_subs 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53940b78e0 con 0x7f539413f710 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 10 monclient: _renew_subs 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53941315f0 con 0x7f539413f710 2024-03-25T19:40:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.897+0000 7f539a93b640 1 librados: init done 2024-03-25T19:40:09.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.898+0000 7f539a93b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.898+0000 7f539a93b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.898+0000 7f539a93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:09.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:09.898+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53940627b0 con 0x7f539413f710 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5388001a50 con 0x7f539413f710 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f5390ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f53880038b0 con 0x7f539413f710 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f5390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f5390ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f5390ff9640 10 monclient: dump: 2024-03-25T19:40:10.161 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:10.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.161+0000 7f53927fc640 4 set_mon_vals no callback set 2024-03-25T19:40:10.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.162+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f5388019900 con 0x7f539413f710 2024-03-25T19:40:10.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.163+0000 7f5393fff640 10 monclient: get_auth_request con 0x7f538006dfb0 auth_method 0 2024-03-25T19:40:10.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.163+0000 7f5390ff9640 10 client.4465.objecter ms_handle_connect 0x7f538006dfb0 2024-03-25T19:40:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.168+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f53880536c0 con 0x7f539413f710 2024-03-25T19:40:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.168+0000 7f5390ff9640 10 client.4465.objecter ms_dispatch 0x7f5394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.168+0000 7f5390ff9640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:10.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.168+0000 7f5390ff9640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:10.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.169+0000 7f5390ff9640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T19:40:10.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.169+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f538804f000 con 0x7f539413f710 2024-03-25T19:40:10.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.169+0000 7f5390ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:10.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.169+0000 7f5390ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:10.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.423+0000 7f539a93b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.423+0000 7f539a93b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.423+0000 7f539a93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:10.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.423+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f539413b1a0 con 0x7f539413f710 2024-03-25T19:40:10.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.548+0000 7f5390ff9640 1 -- 172.21.15.47:0/26189090 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f5388051100 con 0x7f539413f710 2024-03-25T19:40:10.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.548+0000 7f5390ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.548+0000 7f5390ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:10.548 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:10.549 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 10 librados: watch_flush enter 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 10 librados: watch_flush exit 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 10 client.4465.objecter successfully canceled tick 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 >> v1:172.21.15.142:6800/1025302268 conn(0x7f538006dfb0 legacy=0x7f5380070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 10 monclient: shutdown 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 >> v1:172.21.15.142:6789/0 conn(0x7f539413f710 legacy=0x7f53941dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 shutdown_connections 2024-03-25T19:40:10.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 >> 172.21.15.47:0/26189090 conn(0x7f53940b7ea0 msgr2=0x7f5394074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:10.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 shutdown_connections 2024-03-25T19:40:10.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 -- 172.21.15.47:0/26189090 wait complete. 2024-03-25T19:40:10.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:10.551+0000 7f539a93b640 1 librados: shutdown 2024-03-25T19:40:10.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:17.368 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:17.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.470+0000 7fadc27e4640 10 monclient: get_monmap_and_config 2024-03-25T19:40:17.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.470+0000 7fadc27e4640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.470+0000 7fadc27e4640 10 monclient: monmap: 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:17.471010+0000 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:17.471010+0000 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:17.470 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 1 Processor -- start 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 1 -- start start 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 10 monclient: init 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 10 monclient(hunting): picked mon.noname-a con 0x7fadbc13fc60 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 10 monclient(hunting): picked mon.noname-b con 0x7fadbc070570 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:17.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 10 monclient(hunting): picked mon.noname-c con 0x7fadbc0727a0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.471+0000 7fadc27e4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc13c9f0 con 0x7fadbc13fc60 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.472+0000 7fadc27e4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc06fac0 con 0x7fadbc0727a0 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.472+0000 7fadc27e4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc0715d0 con 0x7fadbc070570 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.472+0000 7fadc27e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.472+0000 7fadc27e4640 10 monclient(hunting): authenticate will time out at 5932.757962s 2024-03-25T19:40:17.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.474+0000 7fadc0d81640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fadbc0727a0 0x7fadbc14a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:36628/0 (socket says 172.21.15.47:36628) 2024-03-25T19:40:17.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.475+0000 7fadc0d81640 1 -- 172.21.15.47:0/2638758878 learned_addr learned my addr 172.21.15.47:0/2638758878 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.490+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820386464 0 0) 0x7fadbc06fac0 con 0x7fadbc0727a0 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.490+0000 7fadbaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.490+0000 7fadbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.490+0000 7fadbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.490+0000 7fadbaffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.490+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada4003520 con 0x7fadbc0727a0 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.493+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388166427 0 0) 0x7fadbc0715d0 con 0x7fadbc070570 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.493+0000 7fadbaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.493+0000 7fadbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.493+0000 7fadbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.493+0000 7fadbaffd640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.493+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadbc06fac0 con 0x7fadbc070570 2024-03-25T19:40:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.503+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241148211 0 0) 0x7fadbc13c9f0 con 0x7fadbc13fc60 2024-03-25T19:40:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.503+0000 7fadbaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.503+0000 7fadbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.503+0000 7fadbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.503+0000 7fadbaffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:40:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.503+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadbc0715d0 con 0x7fadbc13fc60 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537229445 0 0) 0x7fadbc06fac0 con 0x7fadbc070570 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 >> v1:172.21.15.142:6789/0 conn(0x7fadbc0727a0 legacy=0x7fadbc14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 >> v1:172.21.15.47:6789/0 conn(0x7fadbc13fc60 legacy=0x7fadbc06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 1 monclient: found mon.noname-b 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc0718b0 con 0x7fadbc070570 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fadb0001670 con 0x7fadbc070570 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadbaffd640 10 monclient: dump: 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadc27e4640 5 monclient: authenticate success, global_id 4460 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.505+0000 7fadc27e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb00017d0 con 0x7fadbc070570 2024-03-25T19:40:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadbaffd640 1 -- 172.21.15.47:0/2638758878 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fadb0001be0 con 0x7fadbc070570 2024-03-25T19:40:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadc27e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadc27e4640 4 set_mon_vals no callback set 2024-03-25T19:40:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadbaffd640 10 monclient: dump: 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:17.564 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadc27e4640 10 monclient: shutdown 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadc27e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.564+0000 7fadc27e4640 1 -- 172.21.15.47:0/2638758878 >> v1:172.21.15.47:6790/0 conn(0x7fadbc070570 legacy=0x7fadbc072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.565+0000 7fadc27e4640 1 -- 172.21.15.47:0/2638758878 shutdown_connections 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.565+0000 7fadc27e4640 1 -- 172.21.15.47:0/2638758878 >> 172.21.15.47:0/2638758878 conn(0x7fadbc0b4a80 msgr2=0x7fadbc0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.565+0000 7fadc27e4640 1 -- 172.21.15.47:0/2638758878 shutdown_connections 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.565+0000 7fadc27e4640 1 -- 172.21.15.47:0/2638758878 wait complete. 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.565+0000 7fadc27e4640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.565+0000 7fadc27e4640 10 monclient: monmap: 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:17.566 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 1 librados: starting msgr at 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 1 librados: starting objecter 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 1 Processor -- start 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 1 -- start start 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 1 librados: setting wanted keys 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 1 librados: calling monclient init 2024-03-25T19:40:17.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.566+0000 7fadc27e4640 10 monclient: init 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient(hunting): picked mon.noname-a con 0x7fadbc070570 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient(hunting): picked mon.noname-b con 0x7fadbc0727a0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient(hunting): picked mon.noname-c con 0x7fadbc13fc60 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc140f60 con 0x7fadbc070570 2024-03-25T19:40:17.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc139c80 con 0x7fadbc0727a0 2024-03-25T19:40:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadbc133800 con 0x7fadbc13fc60 2024-03-25T19:40:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.567+0000 7fadc27e4640 10 monclient(hunting): authenticate will time out at 5932.853499s 2024-03-25T19:40:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.570+0000 7fadbb7fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fadbc0727a0 0x7fadbc1329e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:58548/0 (socket says 172.21.15.47:58548) 2024-03-25T19:40:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.570+0000 7fadbb7fe640 1 -- 172.21.15.47:0/4020335369 learned_addr learned my addr 172.21.15.47:0/4020335369 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:17.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.571+0000 7fadb8ff9640 10 client.?.objecter ms_handle_connect 0x7fadbc0727a0 2024-03-25T19:40:17.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.571+0000 7fadb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.571+0000 7fadb8ff9640 10 client.?.objecter ms_handle_connect 0x7fadbc13fc60 2024-03-25T19:40:17.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.571+0000 7fadb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.578+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549425755 0 0) 0x7fadbc133800 con 0x7fadbc13fc60 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.578+0000 7fadb8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.578+0000 7fadb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.578+0000 7fadb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.578+0000 7fadb8ff9640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.578+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada8004eb0 con 0x7fadbc13fc60 2024-03-25T19:40:17.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.579+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447543267 0 0) 0x7fadbc139c80 con 0x7fadbc0727a0 2024-03-25T19:40:17.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.579+0000 7fadb8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.579+0000 7fadb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.579+0000 7fadb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.579+0000 7fadb8ff9640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:40:17.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.579+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadbc133800 con 0x7fadbc0727a0 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3756049445 0 0) 0x7fada8004eb0 con 0x7fadbc13fc60 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadbc139c80 con 0x7fadbc13fc60 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fadb4001cd0 con 0x7fadbc13fc60 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:17.589 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.590+0000 7fadb8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.591+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4029878721 0 0) 0x7fadbc133800 con 0x7fadbc0727a0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.591+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada8004eb0 con 0x7fadbc0727a0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.591+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fadb0001670 con 0x7fadbc0727a0 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.591+0000 7fadb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.591+0000 7fadb8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.591+0000 7fadb8ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:17.591 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:17.592 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.595+0000 7fadb8ff9640 10 client.?.objecter ms_handle_connect 0x7fadbc070570 2024-03-25T19:40:17.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.595+0000 7fadb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125983245 0 0) 0x7fadbc139c80 con 0x7fadbc13fc60 2024-03-25T19:40:17.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 >> v1:172.21.15.142:6789/0 conn(0x7fadbc0727a0 legacy=0x7fadbc1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 >> v1:172.21.15.47:6789/0 conn(0x7fadbc070570 legacy=0x7fadbc13a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 1 monclient: found mon.c 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc13a820 con 0x7fadbc13fc60 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadc27e4640 5 monclient: authenticate success, global_id 4463 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadc27e4640 10 monclient: _renew_subs 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadc27e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.596+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc141190 con 0x7fadbc13fc60 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.597+0000 7fadc27e4640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.597+0000 7fadc27e4640 10 monclient: _renew_subs 2024-03-25T19:40:17.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.597+0000 7fadc27e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.597+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadbc136610 con 0x7fadbc13fc60 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.597+0000 7fadc27e4640 1 librados: init done 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.598+0000 7fadc27e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.598+0000 7fadc27e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.598+0000 7fadc27e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:17.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.598+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadbc0627b0 con 0x7fadbc13fc60 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb40020e0 con 0x7fadbc13fc60 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fadb4004d00 con 0x7fadbc13fc60 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadb8ff9640 10 monclient: dump: 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.610 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:17.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.610+0000 7fadba7fc640 4 set_mon_vals no callback set 2024-03-25T19:40:17.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.630+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fadb401acd0 con 0x7fadbc13fc60 2024-03-25T19:40:17.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.631+0000 7fadbbfff640 10 monclient: get_auth_request con 0x7fada806dea0 auth_method 0 2024-03-25T19:40:17.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.631+0000 7fadb8ff9640 10 client.4463.objecter ms_handle_connect 0x7fada806dea0 2024-03-25T19:40:17.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fadb40549d0 con 0x7fadbc13fc60 2024-03-25T19:40:17.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 10 client.4463.objecter ms_dispatch 0x7fadbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:17.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:17.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:17.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-25T19:40:17.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fadb4050310 con 0x7fadbc13fc60 2024-03-25T19:40:17.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:17.921+0000 7fadb8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:18.110 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:55.345 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.127 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:55.363 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.149+0000 7fadc27e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:18.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.149+0000 7fadc27e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:18.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.149+0000 7fadc27e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:18.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.149+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadbc0af160 con 0x7fadbc13fc60 2024-03-25T19:40:18.153 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:55.388 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.158 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:55.348 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.171 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:55.361 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.174 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:55.409 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.204 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:55.440 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.210+0000 7fadb8ff9640 1 -- 172.21.15.47:0/4020335369 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fadb4052410 con 0x7fadbc13fc60 2024-03-25T19:40:18.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.210+0000 7fadb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:18.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.210+0000 7fadb8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:18.210 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:18.210 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:18.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 10 librados: watch_flush enter 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 10 librados: watch_flush exit 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 10 client.4463.objecter successfully canceled tick 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 >> v1:172.21.15.142:6800/1025302268 conn(0x7fada806dea0 legacy=0x7fada8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 10 monclient: shutdown 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 >> v1:172.21.15.47:6790/0 conn(0x7fadbc13fc60 legacy=0x7fadbc1330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 shutdown_connections 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.213+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 >> 172.21.15.47:0/4020335369 conn(0x7fadbc0b4a80 msgr2=0x7fadbc0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.214+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 shutdown_connections 2024-03-25T19:40:18.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.214+0000 7fadc27e4640 1 -- 172.21.15.47:0/4020335369 wait complete. 2024-03-25T19:40:18.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:18.214+0000 7fadc27e4640 1 librados: shutdown 2024-03-25T19:40:18.215 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:05:55.404 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:18.227 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:55.463 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.033 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:25.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.151+0000 7f75847c6640 10 monclient: get_monmap_and_config 2024-03-25T19:40:25.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.151+0000 7f75847c6640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.151+0000 7f75847c6640 10 monclient: monmap: 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:25.151976+0000 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:25.151976+0000 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:25.151 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.152+0000 7f75847c6640 1 Processor -- start 2024-03-25T19:40:25.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.152+0000 7f75847c6640 1 -- start start 2024-03-25T19:40:25.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.152+0000 7f75847c6640 10 monclient: init 2024-03-25T19:40:25.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient(hunting): picked mon.noname-b con 0x7f757c06ab80 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient(hunting): picked mon.noname-a con 0x7f757c06af60 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient(hunting): picked mon.noname-c con 0x7f757c0b00b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0b0500 con 0x7f757c06af60 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c13ffa0 con 0x7f757c0b00b0 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0af520 con 0x7f757c06ab80 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.153+0000 7f75847c6640 10 monclient(hunting): authenticate will time out at 5940.439317s 2024-03-25T19:40:25.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.165+0000 7f7582d63640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f757c0b00b0 0x7f757c14dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:35272/0 (socket says 172.21.15.47:35272) 2024-03-25T19:40:25.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.165+0000 7f7582d63640 1 -- 172.21.15.47:0/2480140709 learned_addr learned my addr 172.21.15.47:0/2480140709 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:25.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.173+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930208176 0 0) 0x7f757c13ffa0 con 0x7f757c0b00b0 2024-03-25T19:40:25.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.173+0000 7f7581560640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.173+0000 7f7581560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.173+0000 7f7581560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.173+0000 7f7581560640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:40:25.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.173+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7568003520 con 0x7f757c0b00b0 2024-03-25T19:40:25.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.177+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507097174 0 0) 0x7f757c0af520 con 0x7f757c06ab80 2024-03-25T19:40:25.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.177+0000 7f7581560640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.177+0000 7f7581560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.177+0000 7f7581560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.177+0000 7f7581560640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:40:25.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.177+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c13ffa0 con 0x7f757c06ab80 2024-03-25T19:40:25.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.180+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180773070 0 0) 0x7f757c0b0500 con 0x7f757c06af60 2024-03-25T19:40:25.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.180+0000 7f7581560640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.180+0000 7f7581560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.180+0000 7f7581560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.180+0000 7f7581560640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:40:25.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.180+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c0af520 con 0x7f757c06af60 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2072597381 0 0) 0x7f757c13ffa0 con 0x7f757c06ab80 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 >> v1:172.21.15.142:6789/0 conn(0x7f757c0b00b0 legacy=0x7f757c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 >> v1:172.21.15.47:6789/0 conn(0x7f757c06af60 legacy=0x7f757c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 1 monclient: found mon.noname-b 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c072890 con 0x7f757c06ab80 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f7578001670 con 0x7f757c06ab80 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f7581560640 10 monclient: dump: 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.191 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:25.192 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:25.192 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:25.192 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f75847c6640 5 monclient: authenticate success, global_id 4466 2024-03-25T19:40:25.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.187+0000 7f75847c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75780017d0 con 0x7f757c06ab80 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f7581560640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f7581560640 1 -- 172.21.15.47:0/2480140709 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f7578001be0 con 0x7f757c06ab80 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f75847c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f75847c6640 4 set_mon_vals no callback set 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f7581560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f7581560640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f7581560640 10 monclient: dump: 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:25.203 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f75847c6640 10 monclient: shutdown 2024-03-25T19:40:25.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f75847c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.203+0000 7f75847c6640 1 -- 172.21.15.47:0/2480140709 >> v1:172.21.15.47:6790/0 conn(0x7f757c06ab80 legacy=0x7f757c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.204+0000 7f75847c6640 1 -- 172.21.15.47:0/2480140709 shutdown_connections 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.204+0000 7f75847c6640 1 -- 172.21.15.47:0/2480140709 >> 172.21.15.47:0/2480140709 conn(0x7f757c0b4a80 msgr2=0x7f757c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.204+0000 7f75847c6640 1 -- 172.21.15.47:0/2480140709 shutdown_connections 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.204+0000 7f75847c6640 1 -- 172.21.15.47:0/2480140709 wait complete. 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.204+0000 7f75847c6640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.204+0000 7f75847c6640 10 monclient: monmap: 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:25.206 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 1 librados: starting msgr at 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 1 librados: starting objecter 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 1 Processor -- start 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 1 -- start start 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 1 librados: setting wanted keys 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 1 librados: calling monclient init 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 10 monclient: init 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.205+0000 7f75847c6640 10 monclient(hunting): picked mon.noname-b con 0x7f757c06ab80 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 10 monclient(hunting): picked mon.noname-a con 0x7f757c0b00b0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 10 monclient(hunting): picked mon.noname-c con 0x7f757c1453c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:25.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c145010 con 0x7f757c0b00b0 2024-03-25T19:40:25.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c143200 con 0x7f757c06ab80 2024-03-25T19:40:25.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c143940 con 0x7f757c1453c0 2024-03-25T19:40:25.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.206+0000 7f75847c6640 10 monclient(hunting): authenticate will time out at 5940.492014s 2024-03-25T19:40:25.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.211+0000 7f7581d61640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f757c0b00b0 0x7f757c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47990/0 (socket says 172.21.15.47:47990) 2024-03-25T19:40:25.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.211+0000 7f7581d61640 1 -- 172.21.15.47:0/4202432064 learned_addr learned my addr 172.21.15.47:0/4202432064 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:25.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.212+0000 7f7572ffd640 10 client.?.objecter ms_handle_connect 0x7f757c0b00b0 2024-03-25T19:40:25.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.212+0000 7f7572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.217+0000 7f7572ffd640 10 client.?.objecter ms_handle_connect 0x7f757c1453c0 2024-03-25T19:40:25.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.217+0000 7f7572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.219+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243706784 0 0) 0x7f757c145010 con 0x7f757c0b00b0 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.219+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.219+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.219+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.219+0000 7f7572ffd640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.219+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7560004eb0 con 0x7f757c0b00b0 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.220+0000 7f7572ffd640 10 client.?.objecter ms_handle_connect 0x7f757c06ab80 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.220+0000 7f7572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.224+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275045046 0 0) 0x7f757c143200 con 0x7f757c06ab80 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.224+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.224+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.224+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.224+0000 7f7572ffd640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.224+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c145010 con 0x7f757c06ab80 2024-03-25T19:40:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.228+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407985129 0 0) 0x7f757c143940 con 0x7f757c1453c0 2024-03-25T19:40:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.228+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.228+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.228+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.228+0000 7f7572ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:40:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.228+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c143200 con 0x7f757c1453c0 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878281988 0 0) 0x7f7560004eb0 con 0x7f757c0b00b0 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f757c143940 con 0x7f757c0b00b0 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f756c001a10 con 0x7f757c0b00b0 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.232 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.233 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.233 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:25.233 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:25.233 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:25.233 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.232+0000 7f7572ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:25.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.235+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42356658 0 0) 0x7f757c143200 con 0x7f757c1453c0 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.235+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7560004eb0 con 0x7f757c1453c0 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.235+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f7574001a30 con 0x7f757c1453c0 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.235+0000 7f7572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.235+0000 7f7572ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.235+0000 7f7572ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:25.235 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:25.236 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.251+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189389340 0 0) 0x7f757c145010 con 0x7f757c06ab80 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.251+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f757c143200 con 0x7f757c06ab80 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.251+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f7578001ae0 con 0x7f757c06ab80 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.251+0000 7f7572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.251+0000 7f7572ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.251+0000 7f7572ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.251 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.252 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.252 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.252 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:25.252 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:25.252 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:25.252 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 20092271 0 0) 0x7f7560004eb0 con 0x7f757c1453c0 2024-03-25T19:40:25.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 >> v1:172.21.15.142:6789/0 conn(0x7f757c06ab80 legacy=0x7f757c142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 >> v1:172.21.15.47:6789/0 conn(0x7f757c0b00b0 legacy=0x7f757c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 1 monclient: found mon.c 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c143c40 con 0x7f757c1453c0 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f7572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 10 monclient: _renew_subs 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c1d7df0 con 0x7f757c1453c0 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:25.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 10 monclient: _renew_subs 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c1384c0 con 0x7f757c1453c0 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.253+0000 7f75847c6640 1 librados: init done 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.254+0000 7f75847c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.254+0000 7f75847c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.254+0000 7f75847c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:25.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.254+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c0627b0 con 0x7f757c1453c0 2024-03-25T19:40:25.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7574001f20 con 0x7f757c1453c0 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7572ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f7574004d00 con 0x7f757c1453c0 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7572ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7572ffd640 10 monclient: dump: 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.575 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.575+0000 7f7580d5f640 4 set_mon_vals no callback set 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.576+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f757401acd0 con 0x7f757c1453c0 2024-03-25T19:40:25.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.577+0000 7f7582562640 10 monclient: get_auth_request con 0x7f756006e160 auth_method 0 2024-03-25T19:40:25.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.577+0000 7f7572ffd640 10 client.4469.objecter ms_handle_connect 0x7f756006e160 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.581+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f7574054860 con 0x7f757c1453c0 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.581+0000 7f7572ffd640 10 client.4469.objecter ms_dispatch 0x7f757c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.581+0000 7f7572ffd640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.581+0000 7f7572ffd640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.582+0000 7f7572ffd640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.582+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7574050370 con 0x7f757c1453c0 2024-03-25T19:40:25.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.582+0000 7f7572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.582+0000 7f7572ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:25.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.814+0000 7f75847c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.814+0000 7f75847c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.814+0000 7f75847c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:25.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.814+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f757c143dd0 con 0x7f757c1453c0 2024-03-25T19:40:25.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.841+0000 7f7572ffd640 1 -- 172.21.15.47:0/4202432064 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f75740522a0 con 0x7f757c1453c0 2024-03-25T19:40:25.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.841+0000 7f7572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.841+0000 7f7572ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:25.841 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:25.841 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.843+0000 7f75847c6640 10 librados: watch_flush enter 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.843+0000 7f75847c6640 10 librados: watch_flush exit 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.843+0000 7f75847c6640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.843+0000 7f75847c6640 10 client.4469.objecter successfully canceled tick 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 >> v1:172.21.15.142:6800/1025302268 conn(0x7f756006e160 legacy=0x7f7560070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 10 monclient: shutdown 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 >> v1:172.21.15.47:6790/0 conn(0x7f757c1453c0 legacy=0x7f757c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 shutdown_connections 2024-03-25T19:40:25.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 >> 172.21.15.47:0/4202432064 conn(0x7f757c0b4a80 msgr2=0x7f757c0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 shutdown_connections 2024-03-25T19:40:25.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 -- 172.21.15.47:0/4202432064 wait complete. 2024-03-25T19:40:25.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:25.844+0000 7f75847c6640 1 librados: shutdown 2024-03-25T19:40:25.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:32.663 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:32.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.784+0000 7fdbed293640 10 monclient: get_monmap_and_config 2024-03-25T19:40:32.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.784+0000 7fdbed293640 10 monclient: build_initial_monmap 2024-03-25T19:40:32.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.784+0000 7fdbed293640 10 monclient: monmap: 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:32.786020+0000 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:32.786020+0000 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:32.785 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 1 Processor -- start 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 1 -- start start 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 10 monclient: init 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe80b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.785+0000 7fdbed293640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe80b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.786+0000 7fdbed293640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe814be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.786+0000 7fdbed293640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80af940 con 0x7fdbe80b1730 2024-03-25T19:40:32.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.786+0000 7fdbed293640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80b0a70 con 0x7fdbe80b4d60 2024-03-25T19:40:32.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.786+0000 7fdbed293640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80b0d50 con 0x7fdbe814be90 2024-03-25T19:40:32.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.786+0000 7fdbed293640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:32.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.786+0000 7fdbed293640 10 monclient(hunting): authenticate will time out at 5948.073050s 2024-03-25T19:40:32.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.795+0000 7fdbdffff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbe80b1730 0x7fdbe80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35698/0 (socket says 172.21.15.47:35698) 2024-03-25T19:40:32.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.795+0000 7fdbdffff640 1 -- 172.21.15.47:0/1265033670 learned_addr learned my addr 172.21.15.47:0/1265033670 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:32.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.808+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221829235 0 0) 0x7fdbe80af940 con 0x7fdbe80b1730 2024-03-25T19:40:32.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.808+0000 7fdbe67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.808+0000 7fdbe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.808+0000 7fdbe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.808+0000 7fdbe67fc640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:40:32.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.808+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd4003520 con 0x7fdbe80b1730 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808963004 0 0) 0x7fdbe80b0a70 con 0x7fdbe80b4d60 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe80af940 con 0x7fdbe80b4d60 2024-03-25T19:40:32.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3439014479 0 0) 0x7fdbd4003520 con 0x7fdbe80b1730 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 >> v1:172.21.15.47:6790/0 conn(0x7fdbe814be90 legacy=0x7fdbe814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 >> v1:172.21.15.142:6789/0 conn(0x7fdbe80b4d60 legacy=0x7fdbe813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe80b1030 con 0x7fdbe80b1730 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdbd0001690 con 0x7fdbe80b1730 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbed293640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:40:32.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbed293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbe67fc640 10 monclient: dump: 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.828+0000 7fdbed293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:32.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd00017f0 con 0x7fdbe80b1730 2024-03-25T19:40:32.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbe67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbe67fc640 1 -- 172.21.15.47:0/1265033670 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdbd0001c00 con 0x7fdbe80b1730 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbe67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbe67fc640 10 monclient: dump: 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbed293640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:32.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.843+0000 7fdbed293640 4 set_mon_vals no callback set 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 10 monclient: shutdown 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 1 -- 172.21.15.47:0/1265033670 >> v1:172.21.15.47:6789/0 conn(0x7fdbe80b1730 legacy=0x7fdbe80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 1 -- 172.21.15.47:0/1265033670 shutdown_connections 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 1 -- 172.21.15.47:0/1265033670 >> 172.21.15.47:0/1265033670 conn(0x7fdbe80617a0 msgr2=0x7fdbe8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 1 -- 172.21.15.47:0/1265033670 shutdown_connections 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.844+0000 7fdbed293640 1 -- 172.21.15.47:0/1265033670 wait complete. 2024-03-25T19:40:32.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 10 monclient: build_initial_monmap 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 10 monclient: monmap: 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 1 librados: starting msgr at 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 1 librados: starting objecter 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 1 Processor -- start 2024-03-25T19:40:32.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 1 -- start start 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 1 librados: setting wanted keys 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 1 librados: calling monclient init 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.845+0000 7fdbed293640 10 monclient: init 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe80b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe80b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe814be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe81406d0 con 0x7fdbe80b4d60 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe8137ae0 con 0x7fdbe814be90 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe8140380 con 0x7fdbe80b1730 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:32.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.846+0000 7fdbed293640 10 monclient(hunting): authenticate will time out at 5948.133386s 2024-03-25T19:40:32.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.854+0000 7fdbdffff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbe80b4d60 0x7fdbe8143fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35758/0 (socket says 172.21.15.47:35758) 2024-03-25T19:40:32.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.854+0000 7fdbdffff640 1 -- 172.21.15.47:0/3165037663 learned_addr learned my addr 172.21.15.47:0/3165037663 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:32.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.856+0000 7fdbdf7fe640 10 client.?.objecter ms_handle_connect 0x7fdbe80b4d60 2024-03-25T19:40:32.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.856+0000 7fdbdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:32.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.859+0000 7fdbdf7fe640 10 client.?.objecter ms_handle_connect 0x7fdbe814be90 2024-03-25T19:40:32.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.859+0000 7fdbdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:32.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.862+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459918130 0 0) 0x7fdbe81406d0 con 0x7fdbe80b4d60 2024-03-25T19:40:32.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.862+0000 7fdbdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.862+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.862+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.862+0000 7fdbdf7fe640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:40:32.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.862+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc8004eb0 con 0x7fdbe80b4d60 2024-03-25T19:40:32.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.865+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674926569 0 0) 0x7fdbe8137ae0 con 0x7fdbe814be90 2024-03-25T19:40:32.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.865+0000 7fdbdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.865+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.865+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.865+0000 7fdbdf7fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:40:32.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.865+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe81406d0 con 0x7fdbe814be90 2024-03-25T19:40:32.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.870+0000 7fdbdf7fe640 10 client.?.objecter ms_handle_connect 0x7fdbe80b1730 2024-03-25T19:40:32.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.870+0000 7fdbdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607242118 0 0) 0x7fdbc8004eb0 con 0x7fdbe80b4d60 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe8137ae0 con 0x7fdbe80b4d60 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdbd0001bc0 con 0x7fdbe80b4d60 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:32.876 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.875+0000 7fdbdf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.876+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796490036 0 0) 0x7fdbe8140380 con 0x7fdbe80b1730 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.876+0000 7fdbdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.876+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.876+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:32.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.876+0000 7fdbdf7fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:40:32.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.876+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc8004eb0 con 0x7fdbe80b1730 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.882+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393096217 0 0) 0x7fdbe81406d0 con 0x7fdbe814be90 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.882+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe8140380 con 0x7fdbe814be90 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.882+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdbe0001b90 con 0x7fdbe814be90 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.882+0000 7fdbdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.882+0000 7fdbdf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.882+0000 7fdbdf7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.883 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.883+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491253994 0 0) 0x7fdbc8004eb0 con 0x7fdbe80b1730 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.883+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe81406d0 con 0x7fdbe80b1730 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.883+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdbd8001930 con 0x7fdbe80b1730 2024-03-25T19:40:32.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.883+0000 7fdbdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.883+0000 7fdbdf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.883+0000 7fdbdf7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:32.885 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:32.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1353443457 0 0) 0x7fdbe81406d0 con 0x7fdbe80b1730 2024-03-25T19:40:32.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 >> v1:172.21.15.142:6789/0 conn(0x7fdbe814be90 legacy=0x7fdbe81446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 >> v1:172.21.15.47:6789/0 conn(0x7fdbe80b4d60 legacy=0x7fdbe8143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 1 monclient: found mon.c 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8136660 con 0x7fdbe80b1730 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 5 monclient: authenticate success, global_id 4475 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 10 monclient: _renew_subs 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe8136850 con 0x7fdbe80b1730 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 10 monclient: _renew_subs 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:32.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe81e37c0 con 0x7fdbe80b1730 2024-03-25T19:40:32.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.890+0000 7fdbed293640 1 librados: init done 2024-03-25T19:40:32.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.891+0000 7fdbed293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:32.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.891+0000 7fdbed293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:32.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.891+0000 7fdbed293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:32.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:32.891+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe8140510 con 0x7fdbe80b1730 2024-03-25T19:40:33.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd8001d40 con 0x7fdbe80b1730 2024-03-25T19:40:33.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdbd80037b0 con 0x7fdbe80b1730 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 10 monclient: dump: 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:33.204 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:33.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbe5ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:33.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.203+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fdbd80197c0 con 0x7fdbe80b1730 2024-03-25T19:40:33.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.204+0000 7fdbdffff640 10 monclient: get_auth_request con 0x7fdbc806e100 auth_method 0 2024-03-25T19:40:33.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.204+0000 7fdbdf7fe640 10 client.4475.objecter ms_handle_connect 0x7fdbc806e100 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fdbd8053520 con 0x7fdbe80b1730 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 10 client.4475.objecter ms_dispatch 0x7fdbe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbd804ee60 con 0x7fdbe80b1730 2024-03-25T19:40:33.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:33.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.210+0000 7fdbdf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:33.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.457+0000 7fdbed293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:33.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.457+0000 7fdbed293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:33.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.457+0000 7fdbed293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:33.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.457+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbe80b73e0 con 0x7fdbe80b1730 2024-03-25T19:40:33.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.518+0000 7fdbdf7fe640 1 -- 172.21.15.47:0/3165037663 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fdbd8050f60 con 0x7fdbe80b1730 2024-03-25T19:40:33.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.518+0000 7fdbdf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:33.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.518+0000 7fdbdf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:33.519 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:33.519 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:33.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 10 librados: watch_flush enter 2024-03-25T19:40:33.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 10 librados: watch_flush exit 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 10 client.4475.objecter successfully canceled tick 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 >> v1:172.21.15.142:6800/1025302268 conn(0x7fdbc806e100 legacy=0x7fdbc80705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 10 monclient: shutdown 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.520+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 >> v1:172.21.15.47:6790/0 conn(0x7fdbe80b1730 legacy=0x7fdbe81372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.521+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 shutdown_connections 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.521+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 >> 172.21.15.47:0/3165037663 conn(0x7fdbe80617a0 msgr2=0x7fdbe8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.521+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 shutdown_connections 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.521+0000 7fdbed293640 1 -- 172.21.15.47:0/3165037663 wait complete. 2024-03-25T19:40:33.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:33.521+0000 7fdbed293640 1 librados: shutdown 2024-03-25T19:40:33.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:38.124 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:15.359 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.142 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:15.377 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.166 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:15.401 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.189 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:15.379 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.196 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:15.431 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.197 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:15.387 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.212 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:15.448 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.227 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:15.417 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.244 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:15.480 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:40.339 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.456+0000 7f66fd30e640 10 monclient: get_monmap_and_config 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.456+0000 7f66fd30e640 10 monclient: build_initial_monmap 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.456+0000 7f66fd30e640 10 monclient: monmap: 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:40.456886+0000 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:40.456886+0000 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:40.456 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:40.457 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:40.457 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:40.457 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.456+0000 7f66fd30e640 1 Processor -- start 2024-03-25T19:40:40.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.456+0000 7f66fd30e640 1 -- start start 2024-03-25T19:40:40.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.456+0000 7f66fd30e640 10 monclient: init 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient(hunting): picked mon.noname-a con 0x7f66f80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient(hunting): picked mon.noname-c con 0x7f66f813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient(hunting): picked mon.noname-b con 0x7f66f81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f8142520 con 0x7f66f80a98f0 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80b28a0 con 0x7f66f813f710 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80b2a30 con 0x7f66f81430c0 2024-03-25T19:40:40.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:40.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.457+0000 7f66fd30e640 10 monclient(hunting): authenticate will time out at 5955.743714s 2024-03-25T19:40:40.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.461+0000 7f66f67fc640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f66f813f710 0x7f66f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:46058/0 (socket says 172.21.15.47:46058) 2024-03-25T19:40:40.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.461+0000 7f66f67fc640 1 -- 172.21.15.47:0/3983729052 learned_addr learned my addr 172.21.15.47:0/3983729052 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:40.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.475+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089687344 0 0) 0x7f66f80b28a0 con 0x7f66f813f710 2024-03-25T19:40:40.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.475+0000 7f66f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.475+0000 7f66f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.475+0000 7f66f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.475+0000 7f66f5ffb640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:40:40.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.475+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e0003520 con 0x7f66f813f710 2024-03-25T19:40:40.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3371634950 0 0) 0x7f66e0003520 con 0x7f66f813f710 2024-03-25T19:40:40.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 >> v1:172.21.15.47:6790/0 conn(0x7f66f81430c0 legacy=0x7f66f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 >> v1:172.21.15.47:6789/0 conn(0x7f66f80a98f0 legacy=0x7f66f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f80b44c0 con 0x7f66f813f710 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f66ec001690 con 0x7f66f813f710 2024-03-25T19:40:40.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66f5ffb640 10 monclient: dump: 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66fd30e640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:40:40.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.496+0000 7f66fd30e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66ec0017f0 con 0x7f66f813f710 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66f5ffb640 1 -- 172.21.15.47:0/3983729052 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f66ec001c00 con 0x7f66f813f710 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66f5ffb640 10 monclient: dump: 2024-03-25T19:40:40.501 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:40.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 4 set_mon_vals no callback set 2024-03-25T19:40:40.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 10 monclient: shutdown 2024-03-25T19:40:40.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:40.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 1 -- 172.21.15.47:0/3983729052 >> v1:172.21.15.142:6789/0 conn(0x7f66f813f710 legacy=0x7f66f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 1 -- 172.21.15.47:0/3983729052 shutdown_connections 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 1 -- 172.21.15.47:0/3983729052 >> 172.21.15.47:0/3983729052 conn(0x7f66f80b7ea0 msgr2=0x7f66f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 1 -- 172.21.15.47:0/3983729052 shutdown_connections 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.502+0000 7f66fd30e640 1 -- 172.21.15.47:0/3983729052 wait complete. 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.503+0000 7f66fd30e640 10 monclient: build_initial_monmap 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.503+0000 7f66fd30e640 10 monclient: monmap: 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:40.505 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 1 librados: starting msgr at 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 1 librados: starting objecter 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 1 Processor -- start 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 1 -- start start 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 1 librados: setting wanted keys 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 1 librados: calling monclient init 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.504+0000 7f66fd30e640 10 monclient: init 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:40.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient(hunting): picked mon.noname-b con 0x7f66f80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient(hunting): picked mon.noname-a con 0x7f66f813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient(hunting): picked mon.noname-c con 0x7f66f81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80b1870 con 0x7f66f813f710 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80ac910 con 0x7f66f80a98f0 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80abb00 con 0x7f66f81430c0 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:40.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.505+0000 7f66fd30e640 10 monclient(hunting): authenticate will time out at 5955.791287s 2024-03-25T19:40:40.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.522+0000 7f66f67fc640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f66f813f710 0x7f66f80ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43372/0 (socket says 172.21.15.47:43372) 2024-03-25T19:40:40.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.522+0000 7f66f6ffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f66f80a98f0 0x7f66f8138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:46090/0 (socket says 172.21.15.47:46090) 2024-03-25T19:40:40.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.522+0000 7f66f67fc640 1 -- 172.21.15.47:0/3976609277 learned_addr learned my addr 172.21.15.47:0/3976609277 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:40.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.522+0000 7f66df7fe640 10 client.?.objecter ms_handle_connect 0x7f66f813f710 2024-03-25T19:40:40.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.522+0000 7f66df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:40.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.524+0000 7f66df7fe640 10 client.?.objecter ms_handle_connect 0x7f66f80a98f0 2024-03-25T19:40:40.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.524+0000 7f66df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:40.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.525+0000 7f66df7fe640 10 client.?.objecter ms_handle_connect 0x7f66f81430c0 2024-03-25T19:40:40.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.525+0000 7f66df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:40.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.529+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722018512 0 0) 0x7f66f80abb00 con 0x7f66f81430c0 2024-03-25T19:40:40.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.529+0000 7f66df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.529+0000 7f66df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.529+0000 7f66df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.529+0000 7f66df7fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:40:40.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.529+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e4004eb0 con 0x7f66f81430c0 2024-03-25T19:40:40.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.535+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651255032 0 0) 0x7f66f80b44c0 con 0x7f66f813f710 2024-03-25T19:40:40.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.535+0000 7f66df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.535+0000 7f66df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.535+0000 7f66df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.535+0000 7f66df7fe640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:40:40.535 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.535+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f80abb00 con 0x7f66f813f710 2024-03-25T19:40:40.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.538+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658768858 0 0) 0x7f66f80ac910 con 0x7f66f80a98f0 2024-03-25T19:40:40.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.538+0000 7f66df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:40.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.538+0000 7f66df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:40.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.538+0000 7f66df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:40.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.538+0000 7f66df7fe640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:40:40.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.538+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f80b44c0 con 0x7f66f80a98f0 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3401435678 0 0) 0x7f66f80abb00 con 0x7f66f813f710 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66f80ac910 con 0x7f66f813f710 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f66ec0019a0 con 0x7f66f813f710 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:40.552 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:40.553 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:40.553 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:40.553 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.552+0000 7f66df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.562+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2189734547 0 0) 0x7f66e4004eb0 con 0x7f66f81430c0 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.562+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66f80abb00 con 0x7f66f81430c0 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.562+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f66f0001b90 con 0x7f66f81430c0 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.562+0000 7f66df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.562+0000 7f66df7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.562+0000 7f66df7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.562 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.563 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:40.563 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:40.563 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:40.563 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:40.563 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2007194161 0 0) 0x7f66f80ac910 con 0x7f66f813f710 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 >> v1:172.21.15.47:6790/0 conn(0x7f66f81430c0 legacy=0x7f66f81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 >> v1:172.21.15.142:6789/0 conn(0x7f66f80a98f0 legacy=0x7f66f8138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 1 monclient: found mon.a 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f80abde0 con 0x7f66f813f710 2024-03-25T19:40:40.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 5 monclient: authenticate success, global_id 4503 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 10 monclient: _renew_subs 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f8138f40 con 0x7f66f813f710 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 10 monclient: _renew_subs 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f81363f0 con 0x7f66f813f710 2024-03-25T19:40:40.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.564+0000 7f66fd30e640 1 librados: init done 2024-03-25T19:40:40.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.565+0000 7f66fd30e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.565+0000 7f66fd30e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.565+0000 7f66fd30e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:40.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.565+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f80627b0 con 0x7f66f813f710 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66ec001bc0 con 0x7f66f813f710 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f66ec004c30 con 0x7f66f813f710 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66df7fe640 10 monclient: dump: 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:40.902 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:40.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.902+0000 7f66f57fa640 4 set_mon_vals no callback set 2024-03-25T19:40:40.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.903+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f66ec01ac00 con 0x7f66f813f710 2024-03-25T19:40:40.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.903+0000 7f66f6ffd640 10 monclient: get_auth_request con 0x7f66e406df70 auth_method 0 2024-03-25T19:40:40.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.904+0000 7f66df7fe640 10 client.4503.objecter ms_handle_connect 0x7f66e406df70 2024-03-25T19:40:40.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.910+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f66ec0546c0 con 0x7f66f813f710 2024-03-25T19:40:40.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.910+0000 7f66df7fe640 10 client.4503.objecter ms_dispatch 0x7f66f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:40.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.910+0000 7f66df7fe640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:40.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.910+0000 7f66df7fe640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:40.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.911+0000 7f66df7fe640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T19:40:40.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.911+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66ec0501d0 con 0x7f66f813f710 2024-03-25T19:40:40.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.911+0000 7f66df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:40.911+0000 7f66df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:41.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.170+0000 7f66fd30e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:41.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.170+0000 7f66fd30e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:41.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.170+0000 7f66fd30e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:40:41.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.170+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66f81434a0 con 0x7f66f813f710 2024-03-25T19:40:41.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.196+0000 7f66df7fe640 1 -- 172.21.15.47:0/3976609277 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f66ec052100 con 0x7f66f813f710 2024-03-25T19:40:41.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.196+0000 7f66df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:41.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.196+0000 7f66df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:41.196 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:41.196 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:41.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.198+0000 7f66fd30e640 10 librados: watch_flush enter 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 10 librados: watch_flush exit 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 10 client.4503.objecter successfully canceled tick 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 >> v1:172.21.15.142:6800/1025302268 conn(0x7f66e406df70 legacy=0x7f66e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 10 monclient: shutdown 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 >> v1:172.21.15.47:6789/0 conn(0x7f66f813f710 legacy=0x7f66f80ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 shutdown_connections 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 >> 172.21.15.47:0/3976609277 conn(0x7f66f80b7ea0 msgr2=0x7f66f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 shutdown_connections 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 -- 172.21.15.47:0/3976609277 wait complete. 2024-03-25T19:40:41.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:41.199+0000 7f66fd30e640 1 librados: shutdown 2024-03-25T19:40:41.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:48.018 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.132+0000 7f62b6a0d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.132+0000 7f62b6a0d640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.132+0000 7f62b6a0d640 10 monclient: monmap: 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:48.133246+0000 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:48.133246+0000 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:48.132 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:48.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.133+0000 7f62b6a0d640 1 Processor -- start 2024-03-25T19:40:48.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.133+0000 7f62b6a0d640 1 -- start start 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.133+0000 7f62b6a0d640 10 monclient: init 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f62b00a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f62b013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f62b01430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b0142520 con 0x7f62b00a98f0 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b00b28a0 con 0x7f62b013f710 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b00b2a30 con 0x7f62b01430c0 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.134+0000 7f62b6a0d640 10 monclient(hunting): authenticate will time out at 5963.420281s 2024-03-25T19:40:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.137+0000 7f62b4faa640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f62b01430c0 0x7f62b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51048/0 (socket says 172.21.15.47:51048) 2024-03-25T19:40:48.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.137+0000 7f62b4faa640 1 -- 172.21.15.47:0/2349298231 learned_addr learned my addr 172.21.15.47:0/2349298231 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:48.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.144+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306080833 0 0) 0x7f62b00b2a30 con 0x7f62b01430c0 2024-03-25T19:40:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.144+0000 7f62aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.144+0000 7f62aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.144+0000 7f62aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.144+0000 7f62aeffd640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:40:48.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.144+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6298003520 con 0x7f62b01430c0 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107613752 0 0) 0x7f6298003520 con 0x7f62b01430c0 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 >> v1:172.21.15.142:6789/0 conn(0x7f62b013f710 legacy=0x7f62b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 >> v1:172.21.15.47:6789/0 conn(0x7f62b00a98f0 legacy=0x7f62b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 1 monclient: found mon.noname-b 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:40:48.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b00b44c0 con 0x7f62b01430c0 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62b6a0d640 5 monclient: authenticate success, global_id 4484 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f62a8001a10 con 0x7f62b01430c0 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62b6a0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62aeffd640 10 monclient: dump: 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:48.152 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:48.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.151+0000 7f62b6a0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62a8001bf0 con 0x7f62b01430c0 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62aeffd640 1 -- 172.21.15.47:0/2349298231 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f62a8002000 con 0x7f62b01430c0 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62b6a0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62b6a0d640 4 set_mon_vals no callback set 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:48.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.153+0000 7f62aeffd640 10 monclient: dump: 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 10 monclient: shutdown 2024-03-25T19:40:48.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 1 -- 172.21.15.47:0/2349298231 >> v1:172.21.15.47:6790/0 conn(0x7f62b01430c0 legacy=0x7f62b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 1 -- 172.21.15.47:0/2349298231 shutdown_connections 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 1 -- 172.21.15.47:0/2349298231 >> 172.21.15.47:0/2349298231 conn(0x7f62b00b7ea0 msgr2=0x7f62b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 1 -- 172.21.15.47:0/2349298231 shutdown_connections 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 1 -- 172.21.15.47:0/2349298231 wait complete. 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 10 monclient: build_initial_monmap 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.154+0000 7f62b6a0d640 10 monclient: monmap: 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:48.158 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 1 librados: starting msgr at 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 1 librados: starting objecter 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 1 Processor -- start 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 1 -- start start 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 1 librados: setting wanted keys 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 1 librados: calling monclient init 2024-03-25T19:40:48.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 10 monclient: init 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.155+0000 7f62b6a0d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f62b00a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f62b013f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f62b01430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b01d7500 con 0x7f62b00a98f0 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b01394a0 con 0x7f62b01430c0 2024-03-25T19:40:48.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b0138b50 con 0x7f62b013f710 2024-03-25T19:40:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.156+0000 7f62b6a0d640 10 monclient(hunting): authenticate will time out at 5963.442176s 2024-03-25T19:40:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.160+0000 7f62b4faa640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f62b01430c0 0x7f62b0134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:58402/0 (socket says 172.21.15.47:58402) 2024-03-25T19:40:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.160+0000 7f62b4faa640 1 -- 172.21.15.47:0/404805158 learned_addr learned my addr 172.21.15.47:0/404805158 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.161+0000 7f62acff9640 10 client.?.objecter ms_handle_connect 0x7f62b01430c0 2024-03-25T19:40:48.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.161+0000 7f62acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.172+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991229271 0 0) 0x7f62b01394a0 con 0x7f62b01430c0 2024-03-25T19:40:48.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.172+0000 7f62acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:48.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.172+0000 7f62acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:48.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.172+0000 7f62acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:48.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.172+0000 7f62acff9640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:40:48.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.172+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f629c004eb0 con 0x7f62b01430c0 2024-03-25T19:40:48.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.176+0000 7f62acff9640 10 client.?.objecter ms_handle_connect 0x7f62b013f710 2024-03-25T19:40:48.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.176+0000 7f62acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183102816 0 0) 0x7f629c004eb0 con 0x7f62b01430c0 2024-03-25T19:40:48.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62b01394a0 con 0x7f62b01430c0 2024-03-25T19:40:48.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f62a8001f40 con 0x7f62b01430c0 2024-03-25T19:40:48.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:48.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 10 monclient(hunting): dump: 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:48.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.179+0000 7f62acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:48.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.186+0000 7f62acff9640 10 client.?.objecter ms_handle_connect 0x7f62b00a98f0 2024-03-25T19:40:48.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.186+0000 7f62acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:48.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1667449956 0 0) 0x7f62b01394a0 con 0x7f62b01430c0 2024-03-25T19:40:48.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 >> v1:172.21.15.47:6790/0 conn(0x7f62b013f710 legacy=0x7f62b01d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 >> v1:172.21.15.47:6789/0 conn(0x7f62b00a98f0 legacy=0x7f62b0138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:48.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 1 monclient: found mon.b 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b01d8380 con 0x7f62b01430c0 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 10 monclient: _renew_subs 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b0139710 con 0x7f62b01430c0 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 10 monclient: _renew_subs 2024-03-25T19:40:48.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:48.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b01d7750 con 0x7f62b01430c0 2024-03-25T19:40:48.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.189+0000 7f62b6a0d640 1 librados: init done 2024-03-25T19:40:48.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.190+0000 7f62b6a0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.190+0000 7f62b6a0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.190+0000 7f62b6a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:48.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.190+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b00627b0 con 0x7f62b01430c0 2024-03-25T19:40:48.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62a8004e10 con 0x7f62b01430c0 2024-03-25T19:40:48.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:48.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f62a80015d0 con 0x7f62b01430c0 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62acff9640 10 monclient: dump: 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:48.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.209+0000 7f62ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:40:48.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.210+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f62a801b010 con 0x7f62b01430c0 2024-03-25T19:40:48.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.211+0000 7f62affff640 10 monclient: get_auth_request con 0x7f629c06dd00 auth_method 0 2024-03-25T19:40:48.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.211+0000 7f62acff9640 10 client.4495.objecter ms_handle_connect 0x7f629c06dd00 2024-03-25T19:40:48.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.213+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f62a8054ca0 con 0x7f62b01430c0 2024-03-25T19:40:48.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.213+0000 7f62acff9640 10 client.4495.objecter ms_dispatch 0x7f62b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:48.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.213+0000 7f62acff9640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:48.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.213+0000 7f62acff9640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:48.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.213+0000 7f62acff9640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T19:40:48.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.488+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62a80505e0 con 0x7f62b01430c0 2024-03-25T19:40:48.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.488+0000 7f62acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:48.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.488+0000 7f62acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:48.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.720+0000 7f62b6a0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:48.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.720+0000 7f62b6a0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:48.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.720+0000 7f62b6a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:40:48.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.720+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62b0069f40 con 0x7f62b01430c0 2024-03-25T19:40:48.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.740+0000 7f62acff9640 1 -- 172.21.15.47:0/404805158 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f62a80526e0 con 0x7f62b01430c0 2024-03-25T19:40:48.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.740+0000 7f62acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:48.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.740+0000 7f62acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:48.740 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:48.740 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 10 librados: watch_flush enter 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 10 librados: watch_flush exit 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 10 client.4495.objecter successfully canceled tick 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 >> v1:172.21.15.142:6800/1025302268 conn(0x7f629c06dd00 legacy=0x7f629c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 10 monclient: shutdown 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 >> v1:172.21.15.142:6789/0 conn(0x7f62b01430c0 legacy=0x7f62b0134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:48.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 shutdown_connections 2024-03-25T19:40:48.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 >> 172.21.15.47:0/404805158 conn(0x7f62b00b7ea0 msgr2=0x7f62b013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:48.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.743+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 shutdown_connections 2024-03-25T19:40:48.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.744+0000 7f62b6a0d640 1 -- 172.21.15.47:0/404805158 wait complete. 2024-03-25T19:40:48.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:48.744+0000 7f62b6a0d640 1 librados: shutdown 2024-03-25T19:40:48.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:55.563 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:55.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.684+0000 7fdd69c55640 10 monclient: get_monmap_and_config 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.684+0000 7fdd69c55640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.684+0000 7fdd69c55640 10 monclient: monmap: 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:40:55.685044+0000 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:40:55.685044+0000 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:55.684 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:40:55.685 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:40:55.685 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 1 Processor -- start 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 1 -- start start 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient: init 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient(hunting): picked mon.noname-c con 0x7fdd640a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient(hunting): picked mon.noname-b con 0x7fdd641430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64142520 con 0x7fdd6413f710 2024-03-25T19:40:55.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640b28a0 con 0x7fdd640a98f0 2024-03-25T19:40:55.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640b2a30 con 0x7fdd641430c0 2024-03-25T19:40:55.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.685+0000 7fdd69c55640 10 monclient(hunting): authenticate will time out at 5970.971926s 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.697+0000 7fdd62ffd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdd6413f710 0x7fdd641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49310/0 (socket says 172.21.15.47:49310) 2024-03-25T19:40:55.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.698+0000 7fdd62ffd640 1 -- 172.21.15.47:0/2951293950 learned_addr learned my addr 172.21.15.47:0/2951293950 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:55.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.707+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882051426 0 0) 0x7fdd64142520 con 0x7fdd6413f710 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.707+0000 7fdd627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.707+0000 7fdd627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.707+0000 7fdd627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.707+0000 7fdd627fc640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.707+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4c003520 con 0x7fdd6413f710 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.708+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914321645 0 0) 0x7fdd640b2a30 con 0x7fdd641430c0 2024-03-25T19:40:55.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.708+0000 7fdd627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.708+0000 7fdd627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.708+0000 7fdd627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.708+0000 7fdd627fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:40:55.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.708+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd64142520 con 0x7fdd641430c0 2024-03-25T19:40:55.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401453767 0 0) 0x7fdd4c003520 con 0x7fdd6413f710 2024-03-25T19:40:55.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 >> v1:172.21.15.47:6790/0 conn(0x7fdd641430c0 legacy=0x7fdd6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 >> v1:172.21.15.142:6789/0 conn(0x7fdd640a98f0 legacy=0x7fdd640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd640b44c0 con 0x7fdd6413f710 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdd58001690 con 0x7fdd6413f710 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd69c55640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd69c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.713+0000 7fdd627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.714+0000 7fdd627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.714+0000 7fdd627fc640 10 monclient: dump: 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.714+0000 7fdd69c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd580017f0 con 0x7fdd6413f710 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd627fc640 1 -- 172.21.15.47:0/2951293950 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdd58001c00 con 0x7fdd6413f710 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd69c55640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd69c55640 4 set_mon_vals no callback set 2024-03-25T19:40:55.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.723+0000 7fdd627fc640 10 monclient: dump: 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:55.723 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 10 monclient: shutdown 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 1 -- 172.21.15.47:0/2951293950 >> v1:172.21.15.47:6789/0 conn(0x7fdd6413f710 legacy=0x7fdd641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 1 -- 172.21.15.47:0/2951293950 shutdown_connections 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 1 -- 172.21.15.47:0/2951293950 >> 172.21.15.47:0/2951293950 conn(0x7fdd640b7ea0 msgr2=0x7fdd640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 1 -- 172.21.15.47:0/2951293950 shutdown_connections 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.724+0000 7fdd69c55640 1 -- 172.21.15.47:0/2951293950 wait complete. 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 10 monclient: monmap: 2024-03-25T19:40:55.726 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 1 librados: starting msgr at 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 1 librados: starting objecter 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 1 Processor -- start 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 1 -- start start 2024-03-25T19:40:55.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 1 librados: setting wanted keys 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 1 librados: calling monclient init 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.725+0000 7fdd69c55640 10 monclient: init 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient(hunting): picked mon.noname-a con 0x7fdd640a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient(hunting): picked mon.noname-c con 0x7fdd641430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64136e60 con 0x7fdd640a98f0 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64135a00 con 0x7fdd6413f710 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd641dc070 con 0x7fdd641430c0 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.726+0000 7fdd69c55640 10 monclient(hunting): authenticate will time out at 5971.012526s 2024-03-25T19:40:55.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.739+0000 7fdd62ffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdd6413f710 0x7fdd641db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:34930/0 (socket says 172.21.15.47:34930) 2024-03-25T19:40:55.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.739+0000 7fdd62ffd640 1 -- 172.21.15.47:0/914718658 learned_addr learned my addr 172.21.15.47:0/914718658 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:40:55.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.740+0000 7fdd4bfff640 10 client.?.objecter ms_handle_connect 0x7fdd640a98f0 2024-03-25T19:40:55.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.740+0000 7fdd4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.740+0000 7fdd4bfff640 10 client.?.objecter ms_handle_connect 0x7fdd6413f710 2024-03-25T19:40:55.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.740+0000 7fdd4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.745+0000 7fdd4bfff640 10 client.?.objecter ms_handle_connect 0x7fdd641430c0 2024-03-25T19:40:55.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.745+0000 7fdd4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.751+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465668623 0 0) 0x7fdd641dc070 con 0x7fdd641430c0 2024-03-25T19:40:55.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.751+0000 7fdd4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.751+0000 7fdd4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.751+0000 7fdd4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.751+0000 7fdd4bfff640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:40:55.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.751+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd50004eb0 con 0x7fdd641430c0 2024-03-25T19:40:55.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.752+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841739819 0 0) 0x7fdd64136e60 con 0x7fdd640a98f0 2024-03-25T19:40:55.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.752+0000 7fdd4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.752+0000 7fdd4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.752+0000 7fdd4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.752+0000 7fdd4bfff640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:40:55.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.752+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd641dc070 con 0x7fdd640a98f0 2024-03-25T19:40:55.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.753+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661726278 0 0) 0x7fdd64135a00 con 0x7fdd6413f710 2024-03-25T19:40:55.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.753+0000 7fdd4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.753+0000 7fdd4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.753+0000 7fdd4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.753+0000 7fdd4bfff640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:40:55.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.753+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd64136e60 con 0x7fdd6413f710 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517493721 0 0) 0x7fdd64136e60 con 0x7fdd6413f710 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd64135a00 con 0x7fdd6413f710 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdd58001b00 con 0x7fdd6413f710 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.764 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.765 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:55.765 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:55.765 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:55.765 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:55.765 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.764+0000 7fdd4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.765+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012252129 0 0) 0x7fdd50004eb0 con 0x7fdd641430c0 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.765+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd64136e60 con 0x7fdd641430c0 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.765+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdd5c001bd0 con 0x7fdd641430c0 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.765+0000 7fdd4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.765+0000 7fdd4bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.765+0000 7fdd4bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:55.766 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:55.767 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:55.767 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:55.767 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.769+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3243069146 0 0) 0x7fdd641dc070 con 0x7fdd640a98f0 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.769+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd50004eb0 con 0x7fdd640a98f0 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.769+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdd540018a0 con 0x7fdd640a98f0 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.769+0000 7fdd4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.769+0000 7fdd4bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.769+0000 7fdd4bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:55.769 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:55.770 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:55.770 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:55.770 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2292227179 0 0) 0x7fdd64136e60 con 0x7fdd641430c0 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 >> v1:172.21.15.142:6789/0 conn(0x7fdd6413f710 legacy=0x7fdd641db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 >> v1:172.21.15.47:6789/0 conn(0x7fdd640a98f0 legacy=0x7fdd64137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 1 monclient: found mon.c 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64137fa0 con 0x7fdd641430c0 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 5 monclient: authenticate success, global_id 4493 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 10 monclient: _renew_subs 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd641370b0 con 0x7fdd641430c0 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 10 monclient: _renew_subs 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd640b6f80 con 0x7fdd641430c0 2024-03-25T19:40:55.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.776+0000 7fdd69c55640 1 librados: init done 2024-03-25T19:40:55.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.777+0000 7fdd69c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.777+0000 7fdd69c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.777+0000 7fdd69c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:55.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:55.777+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd640b7230 con 0x7fdd641430c0 2024-03-25T19:40:56.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd5c0020c0 con 0x7fdd641430c0 2024-03-25T19:40:56.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:56.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fdd5c003b10 con 0x7fdd641430c0 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 10 monclient: dump: 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:40:56.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd61ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:56.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.085+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fdd5c019b20 con 0x7fdd641430c0 2024-03-25T19:40:56.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.086+0000 7fdd637fe640 10 monclient: get_auth_request con 0x7fdd5006e1b0 auth_method 0 2024-03-25T19:40:56.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.086+0000 7fdd4bfff640 10 client.4493.objecter ms_handle_connect 0x7fdd5006e1b0 2024-03-25T19:40:56.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fdd5c0537d0 con 0x7fdd641430c0 2024-03-25T19:40:56.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 10 client.4493.objecter ms_dispatch 0x7fdd64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:56.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:56.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:56.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-25T19:40:56.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd5c04f2e0 con 0x7fdd641430c0 2024-03-25T19:40:56.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:56.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.092+0000 7fdd4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:56.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.357+0000 7fdd69c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.357+0000 7fdd69c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.357+0000 7fdd69c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:40:56.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.358+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd640627b0 con 0x7fdd641430c0 2024-03-25T19:40:56.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.381+0000 7fdd4bfff640 1 -- 172.21.15.47:0/914718658 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fdd5c051210 con 0x7fdd641430c0 2024-03-25T19:40:56.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.381+0000 7fdd4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.381+0000 7fdd4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:56.381 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:40:56.381 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:56.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 10 librados: watch_flush enter 2024-03-25T19:40:56.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 10 librados: watch_flush exit 2024-03-25T19:40:56.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-25T19:40:56.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 10 client.4493.objecter successfully canceled tick 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 >> v1:172.21.15.142:6800/1025302268 conn(0x7fdd5006e1b0 legacy=0x7fdd50070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 10 monclient: shutdown 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 >> v1:172.21.15.47:6790/0 conn(0x7fdd641430c0 legacy=0x7fdd641db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 shutdown_connections 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.383+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 >> 172.21.15.47:0/914718658 conn(0x7fdd640b7ea0 msgr2=0x7fdd6414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.384+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 shutdown_connections 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.384+0000 7fdd69c55640 1 -- 172.21.15.47:0/914718658 wait complete. 2024-03-25T19:40:56.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:40:56.384+0000 7fdd69c55640 1 librados: shutdown 2024-03-25T19:40:56.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:58.189 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:35.425 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.199 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:35.434 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.229 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:35.465 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.248 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:35.438 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.256 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:35.446 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.269 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:35.505 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.274 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:35.510 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.289 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:35.480 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.302 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:35.538 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:03.201 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.316+0000 7ff86334b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.316+0000 7ff86334b640 10 monclient: build_initial_monmap 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.316+0000 7ff86334b640 10 monclient: monmap: 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:03.317533+0000 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:03.317533+0000 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:03.316 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:03.317 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.317+0000 7ff86334b640 1 Processor -- start 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.317+0000 7ff86334b640 1 -- start start 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.317+0000 7ff86334b640 10 monclient: init 2024-03-25T19:41:03.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c0b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient(hunting): picked mon.noname-b con 0x7ff85c0b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient(hunting): picked mon.noname-c con 0x7ff85c14be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0af940 con 0x7ff85c0b4d60 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0b0a70 con 0x7ff85c14be90 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0b0d50 con 0x7ff85c0b1730 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:03.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.318+0000 7ff86334b640 10 monclient(hunting): authenticate will time out at 5978.604711s 2024-03-25T19:41:03.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.324+0000 7ff8618e8640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff85c14be90 0x7ff85c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:43150/0 (socket says 172.21.15.47:43150) 2024-03-25T19:41:03.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.324+0000 7ff8618e8640 1 -- 172.21.15.47:0/3650522633 learned_addr learned my addr 172.21.15.47:0/3650522633 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:03.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.342+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096050653 0 0) 0x7ff85c0b0a70 con 0x7ff85c14be90 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.342+0000 7ff853fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.342+0000 7ff853fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.342+0000 7ff853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.342+0000 7ff853fff640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:41:03.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.342+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff848003520 con 0x7ff85c14be90 2024-03-25T19:41:03.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.363+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973292660 0 0) 0x7ff85c0b0d50 con 0x7ff85c0b1730 2024-03-25T19:41:03.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.363+0000 7ff853fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.363+0000 7ff853fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.363+0000 7ff853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.363+0000 7ff853fff640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:41:03.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.363+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85c0b0a70 con 0x7ff85c0b1730 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911421842 0 0) 0x7ff848003520 con 0x7ff85c14be90 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 >> v1:172.21.15.47:6790/0 conn(0x7ff85c0b1730 legacy=0x7ff85c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 >> v1:172.21.15.47:6789/0 conn(0x7ff85c0b4d60 legacy=0x7ff85c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 1 monclient: found mon.noname-c 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c0b1030 con 0x7ff85c14be90 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:03.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff854001a10 con 0x7ff85c14be90 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff853fff640 10 monclient: dump: 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:03.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff86334b640 5 monclient: authenticate success, global_id 4504 2024-03-25T19:41:03.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.365+0000 7ff86334b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:03.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff854001b70 con 0x7ff85c14be90 2024-03-25T19:41:03.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff853fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff853fff640 1 -- 172.21.15.47:0/3650522633 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff854001f80 con 0x7ff85c14be90 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff86334b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff86334b640 4 set_mon_vals no callback set 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff853fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.375+0000 7ff853fff640 10 monclient: dump: 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:03.375 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:03.376 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:03.376 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:03.376 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 10 monclient: shutdown 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 1 -- 172.21.15.47:0/3650522633 >> v1:172.21.15.142:6789/0 conn(0x7ff85c14be90 legacy=0x7ff85c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 1 -- 172.21.15.47:0/3650522633 shutdown_connections 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 1 -- 172.21.15.47:0/3650522633 >> 172.21.15.47:0/3650522633 conn(0x7ff85c0617a0 msgr2=0x7ff85c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 1 -- 172.21.15.47:0/3650522633 shutdown_connections 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.376+0000 7ff86334b640 1 -- 172.21.15.47:0/3650522633 wait complete. 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 10 monclient: build_initial_monmap 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 10 monclient: monmap: 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:03.378 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 1 librados: starting msgr at 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 1 librados: starting objecter 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 1 Processor -- start 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 1 -- start start 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 1 librados: setting wanted keys 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 1 librados: calling monclient init 2024-03-25T19:41:03.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.377+0000 7ff86334b640 10 monclient: init 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient(hunting): picked mon.noname-b con 0x7ff85c0b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient(hunting): picked mon.noname-c con 0x7ff85c0b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c14be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c1d75d0 con 0x7ff85c14be90 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c141e10 con 0x7ff85c0b1730 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c142080 con 0x7ff85c0b4d60 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:03.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.378+0000 7ff86334b640 10 monclient(hunting): authenticate will time out at 5978.664318s 2024-03-25T19:41:03.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.388+0000 7ff8610e7640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff85c0b1730 0x7ff85c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:43208/0 (socket says 172.21.15.47:43208) 2024-03-25T19:41:03.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.388+0000 7ff8610e7640 1 -- 172.21.15.47:0/1524179796 learned_addr learned my addr 172.21.15.47:0/1524179796 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:03.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.394+0000 7ff851ffb640 10 client.?.objecter ms_handle_connect 0x7ff85c0b1730 2024-03-25T19:41:03.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.394+0000 7ff851ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:03.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.396+0000 7ff851ffb640 10 client.?.objecter ms_handle_connect 0x7ff85c0b4d60 2024-03-25T19:41:03.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.396+0000 7ff851ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:03.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.402+0000 7ff851ffb640 10 client.?.objecter ms_handle_connect 0x7ff85c14be90 2024-03-25T19:41:03.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.402+0000 7ff851ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:03.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.405+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741793787 0 0) 0x7ff85c141e10 con 0x7ff85c0b1730 2024-03-25T19:41:03.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.405+0000 7ff851ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.405+0000 7ff851ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.405+0000 7ff851ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.405+0000 7ff851ffb640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:41:03.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.405+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840004eb0 con 0x7ff85c0b1730 2024-03-25T19:41:03.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.408+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292107588 0 0) 0x7ff85c142080 con 0x7ff85c0b4d60 2024-03-25T19:41:03.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.408+0000 7ff851ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.408+0000 7ff851ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.408+0000 7ff851ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.408+0000 7ff851ffb640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:41:03.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.408+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85c141e10 con 0x7ff85c0b4d60 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 576686265 0 0) 0x7ff840004eb0 con 0x7ff85c0b1730 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff85c142080 con 0x7ff85c0b1730 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff858001a30 con 0x7ff85c0b1730 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:03.415 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:03.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.415+0000 7ff851ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:03.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.420+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630882914 0 0) 0x7ff85c1d75d0 con 0x7ff85c14be90 2024-03-25T19:41:03.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.420+0000 7ff851ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:03.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.420+0000 7ff851ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:03.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.420+0000 7ff851ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:03.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.420+0000 7ff851ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:41:03.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.420+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840004eb0 con 0x7ff85c14be90 2024-03-25T19:41:03.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1569722379 0 0) 0x7ff85c142080 con 0x7ff85c0b1730 2024-03-25T19:41:03.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 >> v1:172.21.15.47:6790/0 conn(0x7ff85c0b4d60 legacy=0x7ff85c1d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 >> v1:172.21.15.47:6789/0 conn(0x7ff85c14be90 legacy=0x7ff85c1daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 1 monclient: found mon.b 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c140d00 con 0x7ff85c0b1730 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff86334b640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff86334b640 10 monclient: _renew_subs 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff86334b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.430+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c135a90 con 0x7ff85c0b1730 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 10 monclient: _renew_subs 2024-03-25T19:41:03.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:03.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c1d83a0 con 0x7ff85c0b1730 2024-03-25T19:41:03.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 1 librados: init done 2024-03-25T19:41:03.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:03.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.431+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c0627b0 con 0x7ff85c0b1730 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff858001c70 con 0x7ff85c0b1730 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff851ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff858003a60 con 0x7ff85c0b1730 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff851ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff851ffb640 10 monclient: dump: 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.700 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:03.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.700+0000 7ff8537fe640 4 set_mon_vals no callback set 2024-03-25T19:41:03.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.701+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ff858003c40 con 0x7ff85c0b1730 2024-03-25T19:41:03.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.702+0000 7ff8608e6640 10 monclient: get_auth_request con 0x7ff84006de00 auth_method 0 2024-03-25T19:41:03.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.702+0000 7ff851ffb640 10 client.4507.objecter ms_handle_connect 0x7ff84006de00 2024-03-25T19:41:03.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7ff8580548c0 con 0x7ff85c0b1730 2024-03-25T19:41:03.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 10 client.4507.objecter ms_dispatch 0x7ff85c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:03.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:03.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:03.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T19:41:03.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8580503d0 con 0x7ff85c0b1730 2024-03-25T19:41:03.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.708+0000 7ff851ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:03.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.934+0000 7ff86334b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.934+0000 7ff86334b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.934+0000 7ff86334b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:03.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.934+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85c0b22c0 con 0x7ff85c0b1730 2024-03-25T19:41:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.975+0000 7ff851ffb640 1 -- 172.21.15.47:0/1524179796 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7ff858052300 con 0x7ff85c0b1730 2024-03-25T19:41:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.975+0000 7ff851ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.975+0000 7ff851ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:03.974 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:03.975 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 10 librados: watch_flush enter 2024-03-25T19:41:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 10 librados: watch_flush exit 2024-03-25T19:41:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T19:41:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 10 client.4507.objecter successfully canceled tick 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 >> v1:172.21.15.142:6800/1025302268 conn(0x7ff84006de00 legacy=0x7ff8400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 10 monclient: shutdown 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 >> v1:172.21.15.142:6789/0 conn(0x7ff85c0b1730 legacy=0x7ff85c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 shutdown_connections 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.977+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 >> 172.21.15.47:0/1524179796 conn(0x7ff85c0617a0 msgr2=0x7ff85c0b8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.978+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 shutdown_connections 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.978+0000 7ff86334b640 1 -- 172.21.15.47:0/1524179796 wait complete. 2024-03-25T19:41:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:03.978+0000 7ff86334b640 1 librados: shutdown 2024-03-25T19:41:03.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:10.796 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:10.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.912+0000 7ff55d4ba640 10 monclient: get_monmap_and_config 2024-03-25T19:41:10.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.912+0000 7ff55d4ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:10.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.912+0000 7ff55d4ba640 10 monclient: monmap: 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:10.913724+0000 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:10.913724+0000 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:10.913 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:10.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.913+0000 7ff55d4ba640 1 Processor -- start 2024-03-25T19:41:10.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.913+0000 7ff55d4ba640 1 -- start start 2024-03-25T19:41:10.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.913+0000 7ff55d4ba640 10 monclient: init 2024-03-25T19:41:10.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:10.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:10.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff5580a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff55813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff5581430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558142520 con 0x7ff55813f710 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580b28a0 con 0x7ff5580a98f0 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580b2a30 con 0x7ff5581430c0 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:10.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.914+0000 7ff55d4ba640 10 monclient(hunting): authenticate will time out at 5986.200535s 2024-03-25T19:41:10.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.929+0000 7ff5577fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff5581430c0 0x7ff55814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:50960/0 (socket says 172.21.15.47:50960) 2024-03-25T19:41:10.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.929+0000 7ff5577fe640 1 -- 172.21.15.47:0/766926315 learned_addr learned my addr 172.21.15.47:0/766926315 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:10.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.940+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654332293 0 0) 0x7ff5580b2a30 con 0x7ff5581430c0 2024-03-25T19:41:10.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.940+0000 7ff5567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:10.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.940+0000 7ff5567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:10.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.940+0000 7ff5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:10.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.940+0000 7ff5567fc640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:41:10.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.940+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544003520 con 0x7ff5581430c0 2024-03-25T19:41:10.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.941+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028239431 0 0) 0x7ff558142520 con 0x7ff55813f710 2024-03-25T19:41:10.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.941+0000 7ff5567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:10.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.941+0000 7ff5567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:10.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.941+0000 7ff5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:10.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.941+0000 7ff5567fc640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:41:10.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.941+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5580b2a30 con 0x7ff55813f710 2024-03-25T19:41:10.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.951+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828305326 0 0) 0x7ff5580b28a0 con 0x7ff5580a98f0 2024-03-25T19:41:10.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.951+0000 7ff5567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:10.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.951+0000 7ff5567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:10.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.951+0000 7ff5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:10.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.951+0000 7ff5567fc640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:41:10.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.951+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff558142520 con 0x7ff5580a98f0 2024-03-25T19:41:10.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374129934 0 0) 0x7ff5580b2a30 con 0x7ff55813f710 2024-03-25T19:41:10.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 >> v1:172.21.15.47:6790/0 conn(0x7ff5581430c0 legacy=0x7ff55814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 >> v1:172.21.15.142:6789/0 conn(0x7ff5580a98f0 legacy=0x7ff5580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:10.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 1 monclient: found mon.noname-a 2024-03-25T19:41:10.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:10.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:41:10.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580b44c0 con 0x7ff55813f710 2024-03-25T19:41:10.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:10.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:10.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:10.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff540001670 con 0x7ff55813f710 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff5567fc640 10 monclient: dump: 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:10.956 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:10.957 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:10.957 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:10.957 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:10.957 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:10.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff55d4ba640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:41:10.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.954+0000 7ff55d4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5400017d0 con 0x7ff55813f710 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff5567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff5567fc640 1 -- 172.21.15.47:0/766926315 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff540001be0 con 0x7ff55813f710 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff5567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff5567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff5567fc640 10 monclient: dump: 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:10.963 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:10.964 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:10.964 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:10.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff55d4ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:10.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.963+0000 7ff55d4ba640 4 set_mon_vals no callback set 2024-03-25T19:41:10.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 10 monclient: shutdown 2024-03-25T19:41:10.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:10.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 1 -- 172.21.15.47:0/766926315 >> v1:172.21.15.47:6789/0 conn(0x7ff55813f710 legacy=0x7ff5581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 1 -- 172.21.15.47:0/766926315 shutdown_connections 2024-03-25T19:41:10.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 1 -- 172.21.15.47:0/766926315 >> 172.21.15.47:0/766926315 conn(0x7ff5580b7ea0 msgr2=0x7ff5580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:10.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 1 -- 172.21.15.47:0/766926315 shutdown_connections 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.964+0000 7ff55d4ba640 1 -- 172.21.15.47:0/766926315 wait complete. 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.965+0000 7ff55d4ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.965+0000 7ff55d4ba640 10 monclient: monmap: 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:10.966 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.965+0000 7ff55d4ba640 1 librados: starting msgr at 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.965+0000 7ff55d4ba640 1 librados: starting objecter 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.965+0000 7ff55d4ba640 1 Processor -- start 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.965+0000 7ff55d4ba640 1 -- start start 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 1 librados: setting wanted keys 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 1 librados: calling monclient init 2024-03-25T19:41:10.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 10 monclient: init 2024-03-25T19:41:10.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff5580a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff55813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff5581430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558135590 con 0x7ff5581430c0 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.966+0000 7ff55d4ba640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5581397f0 con 0x7ff5580a98f0 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.967+0000 7ff55d4ba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580ab9e0 con 0x7ff55813f710 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.967+0000 7ff55d4ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:10.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.967+0000 7ff55d4ba640 10 monclient(hunting): authenticate will time out at 5986.252958s 2024-03-25T19:41:10.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.970+0000 7ff556ffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff5580a98f0 0x7ff558133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:57836/0 (socket says 172.21.15.47:57836) 2024-03-25T19:41:10.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.970+0000 7ff556ffd640 1 -- 172.21.15.47:0/1647012660 learned_addr learned my addr 172.21.15.47:0/1647012660 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:10.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.971+0000 7ff54f7fe640 10 client.?.objecter ms_handle_connect 0x7ff5580a98f0 2024-03-25T19:41:10.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.971+0000 7ff54f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:10.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.973+0000 7ff54f7fe640 10 client.?.objecter ms_handle_connect 0x7ff55813f710 2024-03-25T19:41:10.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.973+0000 7ff54f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:10.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.975+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440447976 0 0) 0x7ff5581397f0 con 0x7ff5580a98f0 2024-03-25T19:41:10.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.975+0000 7ff54f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:10.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.975+0000 7ff54f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:10.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.975+0000 7ff54f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:10.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.975+0000 7ff54f7fe640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:41:10.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.975+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538004eb0 con 0x7ff5580a98f0 2024-03-25T19:41:10.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.981+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307225226 0 0) 0x7ff5580ab9e0 con 0x7ff55813f710 2024-03-25T19:41:10.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.981+0000 7ff54f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:10.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.981+0000 7ff54f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:10.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.981+0000 7ff54f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:10.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.981+0000 7ff54f7fe640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:41:10.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.981+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5581397f0 con 0x7ff55813f710 2024-03-25T19:41:10.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52823740 0 0) 0x7ff5581397f0 con 0x7ff55813f710 2024-03-25T19:41:10.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5580ab9e0 con 0x7ff55813f710 2024-03-25T19:41:10.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff540001ae0 con 0x7ff55813f710 2024-03-25T19:41:10.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:10.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:10.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:10.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.990+0000 7ff54f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:10.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.995+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 671231438 0 0) 0x7ff538004eb0 con 0x7ff5580a98f0 2024-03-25T19:41:10.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.995+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5581397f0 con 0x7ff5580a98f0 2024-03-25T19:41:10.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.995+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff548001a10 con 0x7ff5580a98f0 2024-03-25T19:41:10.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.995+0000 7ff54f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.995+0000 7ff54f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.995+0000 7ff54f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:10.995 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:10.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.999+0000 7ff54f7fe640 10 client.?.objecter ms_handle_connect 0x7ff5581430c0 2024-03-25T19:41:10.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:10.999+0000 7ff54f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3671597086 0 0) 0x7ff5581397f0 con 0x7ff5580a98f0 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 >> v1:172.21.15.47:6790/0 conn(0x7ff55813f710 legacy=0x7ff558135a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 >> v1:172.21.15.47:6789/0 conn(0x7ff5581430c0 legacy=0x7ff558136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 1 monclient: found mon.b 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5581351f0 con 0x7ff5580a98f0 2024-03-25T19:41:11.006 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff54f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 10 monclient: _renew_subs 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558137950 con 0x7ff5580a98f0 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 10 monclient: _renew_subs 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558138740 con 0x7ff5580a98f0 2024-03-25T19:41:11.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.006+0000 7ff55d4ba640 1 librados: init done 2024-03-25T19:41:11.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.007+0000 7ff55d4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:11.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.007+0000 7ff55d4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:11.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.007+0000 7ff55d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:11.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.007+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5580abee0 con 0x7ff5580a98f0 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff548001c70 con 0x7ff5580a98f0 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff54f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff548003a60 con 0x7ff5580a98f0 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff54f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff54f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff54f7fe640 10 monclient: dump: 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:11.296 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.296+0000 7ff555ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:11.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.297+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ff548003c80 con 0x7ff5580a98f0 2024-03-25T19:41:11.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.298+0000 7ff54ffff640 10 monclient: get_auth_request con 0x7ff53806dec0 auth_method 0 2024-03-25T19:41:11.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.298+0000 7ff54f7fe640 10 client.4513.objecter ms_handle_connect 0x7ff53806dec0 2024-03-25T19:41:11.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.302+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7ff5480548d0 con 0x7ff5580a98f0 2024-03-25T19:41:11.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.302+0000 7ff54f7fe640 10 client.4513.objecter ms_dispatch 0x7ff558000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:11.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.302+0000 7ff54f7fe640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:11.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.302+0000 7ff54f7fe640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:11.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.302+0000 7ff54f7fe640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:41:11.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.303+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5480503e0 con 0x7ff5580a98f0 2024-03-25T19:41:11.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.303+0000 7ff54f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:11.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.303+0000 7ff54f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:11.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.533+0000 7ff55d4ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:11.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.533+0000 7ff55d4ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:11.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.533+0000 7ff55d4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:11.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.533+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff55814f2d0 con 0x7ff5580a98f0 2024-03-25T19:41:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.553+0000 7ff54f7fe640 1 -- 172.21.15.47:0/1647012660 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7ff548052310 con 0x7ff5580a98f0 2024-03-25T19:41:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.553+0000 7ff54f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.553+0000 7ff54f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:11.552 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:11.553 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:11.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 10 librados: watch_flush enter 2024-03-25T19:41:11.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 10 librados: watch_flush exit 2024-03-25T19:41:11.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:41:11.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 >> v1:172.21.15.142:6800/1025302268 conn(0x7ff53806dec0 legacy=0x7ff538070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 10 monclient: shutdown 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 >> v1:172.21.15.142:6789/0 conn(0x7ff5580a98f0 legacy=0x7ff558133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 shutdown_connections 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.555+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 >> 172.21.15.47:0/1647012660 conn(0x7ff5580b7ea0 msgr2=0x7ff558074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.556+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 shutdown_connections 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.556+0000 7ff55d4ba640 1 -- 172.21.15.47:0/1647012660 wait complete. 2024-03-25T19:41:11.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:11.556+0000 7ff55d4ba640 1 librados: shutdown 2024-03-25T19:41:11.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:18.255 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:55.491 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.267 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:55.502 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.294 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:55.530 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.310 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:55.500 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.314 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:55.504 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.323 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:55.558 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.333 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:55.568 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.347 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:06:55.537 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.357 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:55.593 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:18.374 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.473+0000 7f12936f4640 10 monclient: get_monmap_and_config 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.473+0000 7f12936f4640 10 monclient: build_initial_monmap 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.473+0000 7f12936f4640 10 monclient: monmap: 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:18.475533+0000 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:18.475533+0000 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:18.474 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:18.475 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:18.475 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:18.475 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:18.475 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.474+0000 7f12936f4640 1 Processor -- start 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.474+0000 7f12936f4640 1 -- start start 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.474+0000 7f12936f4640 10 monclient: init 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient(hunting): picked mon.noname-b con 0x7f128c0b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient(hunting): picked mon.noname-a con 0x7f128c0b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient(hunting): picked mon.noname-c con 0x7f128c14be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0af940 con 0x7f128c0b1730 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0b0a70 con 0x7f128c14be90 2024-03-25T19:41:18.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0b0d50 con 0x7f128c0b4d60 2024-03-25T19:41:18.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:18.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.475+0000 7f12936f4640 10 monclient(hunting): authenticate will time out at 5993.762162s 2024-03-25T19:41:18.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.514+0000 7f1290c8f640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f128c0b1730 0x7f128c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:46130/0 (socket says 172.21.15.47:46130) 2024-03-25T19:41:18.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.514+0000 7f1290c8f640 1 -- 172.21.15.47:0/2242134672 learned_addr learned my addr 172.21.15.47:0/2242134672 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:18.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.542+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949078299 0 0) 0x7f128c0af940 con 0x7f128c0b1730 2024-03-25T19:41:18.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.542+0000 7f1283fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.542+0000 7f1283fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.542+0000 7f1283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.542+0000 7f1283fff640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:41:18.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.542+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003520 con 0x7f128c0b1730 2024-03-25T19:41:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.562+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215180583 0 0) 0x7f128c0b0d50 con 0x7f128c0b4d60 2024-03-25T19:41:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.562+0000 7f1283fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.562+0000 7f1283fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.562+0000 7f1283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.562+0000 7f1283fff640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:41:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.562+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c0af940 con 0x7f128c0b4d60 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927273834 0 0) 0x7f1278003520 con 0x7f128c0b1730 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 >> v1:172.21.15.47:6790/0 conn(0x7f128c0b4d60 legacy=0x7f128c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 >> v1:172.21.15.142:6789/0 conn(0x7f128c14be90 legacy=0x7f128c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 1 monclient: found mon.noname-a 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c0b1030 con 0x7f128c0b1730 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:18.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f127c001690 con 0x7f128c0b1730 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f1283fff640 10 monclient: dump: 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:18.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f12936f4640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:41:18.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.571+0000 7f12936f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:18.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f127c0017f0 con 0x7f128c0b1730 2024-03-25T19:41:18.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f1283fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:18.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f1283fff640 1 -- 172.21.15.47:0/2242134672 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f127c001c00 con 0x7f128c0b1730 2024-03-25T19:41:18.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f1283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f1283fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f1283fff640 10 monclient: dump: 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f12936f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:18.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f12936f4640 4 set_mon_vals no callback set 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f12936f4640 10 monclient: shutdown 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f12936f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.582+0000 7f12936f4640 1 -- 172.21.15.47:0/2242134672 >> v1:172.21.15.47:6789/0 conn(0x7f128c0b1730 legacy=0x7f128c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.583+0000 7f12936f4640 1 -- 172.21.15.47:0/2242134672 shutdown_connections 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.583+0000 7f12936f4640 1 -- 172.21.15.47:0/2242134672 >> 172.21.15.47:0/2242134672 conn(0x7f128c0617a0 msgr2=0x7f128c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.583+0000 7f12936f4640 1 -- 172.21.15.47:0/2242134672 shutdown_connections 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.583+0000 7f12936f4640 1 -- 172.21.15.47:0/2242134672 wait complete. 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 10 monclient: build_initial_monmap 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 10 monclient: monmap: 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:18.585 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 1 librados: starting msgr at 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 1 librados: starting objecter 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 1 Processor -- start 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 1 -- start start 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 1 librados: setting wanted keys 2024-03-25T19:41:18.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 1 librados: calling monclient init 2024-03-25T19:41:18.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.584+0000 7f12936f4640 10 monclient: init 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient(hunting): picked mon.noname-c con 0x7f128c0b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient(hunting): picked mon.noname-b con 0x7f128c0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient(hunting): picked mon.noname-a con 0x7f128c14be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c136450 con 0x7f128c14be90 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c143900 con 0x7f128c0b4d60 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c143a90 con 0x7f128c0b1730 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:18.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.585+0000 7f12936f4640 10 monclient(hunting): authenticate will time out at 5993.872543s 2024-03-25T19:41:18.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.593+0000 7f1291490640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f128c0b1730 0x7f128c135c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56532/0 (socket says 172.21.15.47:56532) 2024-03-25T19:41:18.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.593+0000 7f1291490640 1 -- 172.21.15.47:0/3962842343 learned_addr learned my addr 172.21.15.47:0/3962842343 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:18.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.595+0000 7f1281ffb640 10 client.?.objecter ms_handle_connect 0x7f128c0b1730 2024-03-25T19:41:18.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.595+0000 7f1281ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.604+0000 7f1281ffb640 10 client.?.objecter ms_handle_connect 0x7f128c0b4d60 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.604+0000 7f1281ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.604+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045573447 0 0) 0x7f128c143a90 con 0x7f128c0b1730 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.605+0000 7f1281ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.605+0000 7f1281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.605+0000 7f1281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.605+0000 7f1281ffb640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:41:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.605+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1270004eb0 con 0x7f128c0b1730 2024-03-25T19:41:18.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.607+0000 7f1281ffb640 10 client.?.objecter ms_handle_connect 0x7f128c14be90 2024-03-25T19:41:18.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.607+0000 7f1281ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.611+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500771849 0 0) 0x7f128c143900 con 0x7f128c0b4d60 2024-03-25T19:41:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.611+0000 7f1281ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.611+0000 7f1281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.611+0000 7f1281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.611+0000 7f1281ffb640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:41:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.611+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c143a90 con 0x7f128c0b4d60 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 529677381 0 0) 0x7f1270004eb0 con 0x7f128c0b1730 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f128c143900 con 0x7f128c0b1730 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1288001a10 con 0x7f128c0b1730 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:18.618 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:18.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.617+0000 7f1281ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:18.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.623+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987959153 0 0) 0x7f128c136450 con 0x7f128c14be90 2024-03-25T19:41:18.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.623+0000 7f1281ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:18.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.623+0000 7f1281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:18.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.623+0000 7f1281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:18.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.623+0000 7f1281ffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:41:18.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.623+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1270004eb0 con 0x7f128c14be90 2024-03-25T19:41:18.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3704210945 0 0) 0x7f128c143900 con 0x7f128c0b1730 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 >> v1:172.21.15.142:6789/0 conn(0x7f128c0b4d60 legacy=0x7f128c136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 >> v1:172.21.15.47:6789/0 conn(0x7f128c14be90 legacy=0x7f128c13b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 1 monclient: found mon.c 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c1db8c0 con 0x7f128c0b1730 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f1281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 5 monclient: authenticate success, global_id 4511 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 10 monclient: _renew_subs 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c1dbb10 con 0x7f128c0b1730 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 10 monclient: _renew_subs 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c1372b0 con 0x7f128c0b1730 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.625+0000 7f12936f4640 1 librados: init done 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.626+0000 7f12936f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.626+0000 7f12936f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.626+0000 7f12936f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.626+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128c0627b0 con 0x7f128c0b1730 2024-03-25T19:41:18.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.635+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1288001c50 con 0x7f128c0b1730 2024-03-25T19:41:18.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.635+0000 7f1281ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:18.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.635+0000 7f12837fe640 4 set_mon_vals no callback set 2024-03-25T19:41:18.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.939+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1288003a60 con 0x7f128c0b1730 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.939+0000 7f1281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.939+0000 7f1281ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.939+0000 7f1281ffb640 10 monclient: dump: 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:18.940 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:18.941 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:18.941 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:18.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.940+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f1288003c40 con 0x7f128c0b1730 2024-03-25T19:41:18.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.941+0000 7f1290c8f640 10 monclient: get_auth_request con 0x7f127006de00 auth_method 0 2024-03-25T19:41:18.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.941+0000 7f1281ffb640 10 client.4511.objecter ms_handle_connect 0x7f127006de00 2024-03-25T19:41:18.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f12880548e0 con 0x7f128c0b1730 2024-03-25T19:41:18.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 10 client.4511.objecter ms_dispatch 0x7f128c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:18.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:18.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:18.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-25T19:41:18.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12880503f0 con 0x7f128c0b1730 2024-03-25T19:41:18.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:18.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:18.947+0000 7f1281ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:19.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.209+0000 7f12936f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:19.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.209+0000 7f12936f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:19.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.209+0000 7f12936f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:19.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.209+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f128c069f40 con 0x7f128c0b1730 2024-03-25T19:41:19.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.240+0000 7f1281ffb640 1 -- 172.21.15.47:0/3962842343 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f1288052320 con 0x7f128c0b1730 2024-03-25T19:41:19.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.240+0000 7f1281ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:19.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.240+0000 7f1281ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:19.241 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:19.241 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 10 librados: watch_flush enter 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 10 librados: watch_flush exit 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 10 client.4511.objecter successfully canceled tick 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 >> v1:172.21.15.142:6800/1025302268 conn(0x7f127006de00 legacy=0x7f12700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 10 monclient: shutdown 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 >> v1:172.21.15.47:6790/0 conn(0x7f128c0b1730 legacy=0x7f128c135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:19.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 shutdown_connections 2024-03-25T19:41:19.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 >> 172.21.15.47:0/3962842343 conn(0x7f128c0617a0 msgr2=0x7f128c13c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:19.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 shutdown_connections 2024-03-25T19:41:19.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 -- 172.21.15.47:0/3962842343 wait complete. 2024-03-25T19:41:19.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:19.243+0000 7f12936f4640 1 librados: shutdown 2024-03-25T19:41:19.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:26.064 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:26.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.181+0000 7fb59c9f6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:26.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.181+0000 7fb59c9f6640 10 monclient: build_initial_monmap 2024-03-25T19:41:26.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.181+0000 7fb59c9f6640 10 monclient: monmap: 2024-03-25T19:41:26.181 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:26.181 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:26.181 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:26.182610+0000 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:26.182610+0000 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:26.182 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.182+0000 7fb59c9f6640 1 Processor -- start 2024-03-25T19:41:26.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.182+0000 7fb59c9f6640 1 -- start start 2024-03-25T19:41:26.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.182+0000 7fb59c9f6640 10 monclient: init 2024-03-25T19:41:26.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:26.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:26.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5980a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb59813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5981430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb598142520 con 0x7fb59813f710 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5980b28a0 con 0x7fb5980a98f0 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5980b2a30 con 0x7fb5981430c0 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:26.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.183+0000 7fb59c9f6640 10 monclient(hunting): authenticate will time out at 6001.469791s 2024-03-25T19:41:26.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.199+0000 7fb59659c640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb5980a98f0 0x7fb5980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:54158/0 (socket says 172.21.15.47:54158) 2024-03-25T19:41:26.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.199+0000 7fb59659c640 1 -- 172.21.15.47:0/1983783280 learned_addr learned my addr 172.21.15.47:0/1983783280 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:26.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.207+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553185445 0 0) 0x7fb5980b28a0 con 0x7fb5980a98f0 2024-03-25T19:41:26.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.207+0000 7fb595d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.207+0000 7fb595d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.207+0000 7fb595d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.207+0000 7fb595d9b640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:41:26.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.207+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb584003520 con 0x7fb5980a98f0 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1702239071 0 0) 0x7fb584003520 con 0x7fb5980a98f0 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 >> v1:172.21.15.47:6790/0 conn(0x7fb5981430c0 legacy=0x7fb59814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 >> v1:172.21.15.47:6789/0 conn(0x7fb59813f710 legacy=0x7fb5981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 1 monclient: found mon.noname-c 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980b44c0 con 0x7fb5980a98f0 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:26.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb588001690 con 0x7fb5980a98f0 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb59c9f6640 5 monclient: authenticate success, global_id 4522 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb59c9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb595d9b640 10 monclient: dump: 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:26.222 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.221+0000 7fb59c9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:26.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb588001860 con 0x7fb5980a98f0 2024-03-25T19:41:26.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb595d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb595d9b640 1 -- 172.21.15.47:0/1983783280 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb588001cb0 con 0x7fb5980a98f0 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb595d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb595d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb595d9b640 10 monclient: dump: 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb59c9f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:26.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.236+0000 7fb59c9f6640 4 set_mon_vals no callback set 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 10 monclient: shutdown 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 1 -- 172.21.15.47:0/1983783280 >> v1:172.21.15.142:6789/0 conn(0x7fb5980a98f0 legacy=0x7fb5980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 1 -- 172.21.15.47:0/1983783280 shutdown_connections 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 1 -- 172.21.15.47:0/1983783280 >> 172.21.15.47:0/1983783280 conn(0x7fb5980b7ea0 msgr2=0x7fb5980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 1 -- 172.21.15.47:0/1983783280 shutdown_connections 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.237+0000 7fb59c9f6640 1 -- 172.21.15.47:0/1983783280 wait complete. 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.238+0000 7fb59c9f6640 10 monclient: build_initial_monmap 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.238+0000 7fb59c9f6640 10 monclient: monmap: 2024-03-25T19:41:26.238 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.238+0000 7fb59c9f6640 1 librados: starting msgr at 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.238+0000 7fb59c9f6640 1 librados: starting objecter 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 Processor -- start 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 -- start start 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 librados: setting wanted keys 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 librados: calling monclient init 2024-03-25T19:41:26.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient: init 2024-03-25T19:41:26.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:26.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5980a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb59813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5981430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb598137cf0 con 0x7fb5980a98f0 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb598136760 con 0x7fb59813f710 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5981368f0 con 0x7fb5981430c0 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:26.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.239+0000 7fb59c9f6640 10 monclient(hunting): authenticate will time out at 6001.525613s 2024-03-25T19:41:26.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.256+0000 7fb596d9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb5981430c0 0x7fb5981391e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33430/0 (socket says 172.21.15.47:33430) 2024-03-25T19:41:26.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.256+0000 7fb596d9d640 1 -- 172.21.15.47:0/2539837177 learned_addr learned my addr 172.21.15.47:0/2539837177 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:26.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.257+0000 7fb58effd640 10 client.?.objecter ms_handle_connect 0x7fb5981430c0 2024-03-25T19:41:26.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.257+0000 7fb58effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:26.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.262+0000 7fb58effd640 10 client.?.objecter ms_handle_connect 0x7fb59813f710 2024-03-25T19:41:26.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.262+0000 7fb58effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:26.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.266+0000 7fb58effd640 10 client.?.objecter ms_handle_connect 0x7fb5980a98f0 2024-03-25T19:41:26.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.266+0000 7fb58effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:26.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.274+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300115094 0 0) 0x7fb598136760 con 0x7fb59813f710 2024-03-25T19:41:26.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.274+0000 7fb58effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.274+0000 7fb58effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.274+0000 7fb58effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.274+0000 7fb58effd640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:41:26.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.275+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb578004eb0 con 0x7fb59813f710 2024-03-25T19:41:26.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.277+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537216655 0 0) 0x7fb598137cf0 con 0x7fb5980a98f0 2024-03-25T19:41:26.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.277+0000 7fb58effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.277+0000 7fb58effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.277+0000 7fb58effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.277+0000 7fb58effd640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:41:26.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.277+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598136760 con 0x7fb5980a98f0 2024-03-25T19:41:26.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.287+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309571344 0 0) 0x7fb5981368f0 con 0x7fb5981430c0 2024-03-25T19:41:26.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.287+0000 7fb58effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:26.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.287+0000 7fb58effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:26.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.287+0000 7fb58effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:26.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.287+0000 7fb58effd640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:41:26.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.287+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598137cf0 con 0x7fb5981430c0 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3384028790 0 0) 0x7fb578004eb0 con 0x7fb59813f710 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5981368f0 con 0x7fb59813f710 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb580001870 con 0x7fb59813f710 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 10 monclient(hunting): dump: 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:26.292 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.292+0000 7fb58effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.294+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985202906 0 0) 0x7fb598136760 con 0x7fb5980a98f0 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.294+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb578004eb0 con 0x7fb5980a98f0 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.294+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb588001b70 con 0x7fb5980a98f0 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.294+0000 7fb58effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.294+0000 7fb58effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.294+0000 7fb58effd640 10 monclient(hunting): dump: 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.294 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.295 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.295 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:26.295 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:26.295 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:26.295 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:26.295 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2099642941 0 0) 0x7fb578004eb0 con 0x7fb5980a98f0 2024-03-25T19:41:26.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 >> v1:172.21.15.47:6790/0 conn(0x7fb5981430c0 legacy=0x7fb5981391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 >> v1:172.21.15.142:6789/0 conn(0x7fb59813f710 legacy=0x7fb598137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 1 monclient: found mon.a 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59813b3c0 con 0x7fb5980a98f0 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 10 monclient: _finish_auth 0 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.300+0000 7fb58effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 10 monclient: _renew_subs 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59813b630 con 0x7fb5980a98f0 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 10 monclient: _renew_subs 2024-03-25T19:41:26.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:41:26.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb598133150 con 0x7fb5980a98f0 2024-03-25T19:41:26.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.301+0000 7fb59c9f6640 1 librados: init done 2024-03-25T19:41:26.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.302+0000 7fb59c9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:26.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.302+0000 7fb59c9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:26.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.302+0000 7fb59c9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:41:26.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.302+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5980b3ef0 con 0x7fb5980a98f0 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb588001cd0 con 0x7fb5980a98f0 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb58effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb588004d90 con 0x7fb5980a98f0 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb58effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb58effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb58effd640 10 monclient: dump: 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.631 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.631+0000 7fb59559a640 4 set_mon_vals no callback set 2024-03-25T19:41:26.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.632+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fb58801ad60 con 0x7fb5980a98f0 2024-03-25T19:41:26.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.633+0000 7fb58ffff640 10 monclient: get_auth_request con 0x7fb57806dfb0 auth_method 0 2024-03-25T19:41:26.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.633+0000 7fb58effd640 10 client.4536.objecter ms_handle_connect 0x7fb57806dfb0 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fb588054800 con 0x7fb5980a98f0 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 10 client.4536.objecter ms_dispatch 0x7fb598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb588050310 con 0x7fb5980a98f0 2024-03-25T19:41:26.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:26.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.638+0000 7fb58effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:26.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.859+0000 7fb59c9f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:26.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.859+0000 7fb59c9f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:26.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.859+0000 7fb59c9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:41:26.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.859+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5980627b0 con 0x7fb5980a98f0 2024-03-25T19:41:26.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.942+0000 7fb58effd640 1 -- 172.21.15.47:0/2539837177 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fb588052240 con 0x7fb5980a98f0 2024-03-25T19:41:26.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.942+0000 7fb58effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:26.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.942+0000 7fb58effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:26.942 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:26.943 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:26.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 10 librados: watch_flush enter 2024-03-25T19:41:26.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 10 librados: watch_flush exit 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 10 client.4536.objecter successfully canceled tick 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 >> v1:172.21.15.142:6800/1025302268 conn(0x7fb57806dfb0 legacy=0x7fb578070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 10 monclient: shutdown 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 >> v1:172.21.15.47:6789/0 conn(0x7fb5980a98f0 legacy=0x7fb5981da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 shutdown_connections 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.945+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 >> 172.21.15.47:0/2539837177 conn(0x7fb5980b7ea0 msgr2=0x7fb59813c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.946+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 shutdown_connections 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.946+0000 7fb59c9f6640 1 -- 172.21.15.47:0/2539837177 wait complete. 2024-03-25T19:41:26.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:26.946+0000 7fb59c9f6640 1 librados: shutdown 2024-03-25T19:41:26.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:33.764 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.880+0000 7f050570e640 10 monclient: get_monmap_and_config 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.880+0000 7f050570e640 10 monclient: build_initial_monmap 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.880+0000 7f050570e640 10 monclient: monmap: 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:33.881609+0000 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:33.881609+0000 2024-03-25T19:41:33.880 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:33.881 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:33.881 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:33.881 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:33.881 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:33.881 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.881+0000 7f050570e640 1 Processor -- start 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.881+0000 7f050570e640 1 -- start start 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.881+0000 7f050570e640 10 monclient: init 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient(hunting): picked mon.noname-b con 0x7f050006ea40 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:33.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient(hunting): picked mon.noname-a con 0x7f0500071800 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:33.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient(hunting): picked mon.noname-c con 0x7f0500071c40 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:33.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050006f1f0 con 0x7f0500071800 2024-03-25T19:41:33.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500070d00 con 0x7f0500071c40 2024-03-25T19:41:33.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000720b0 con 0x7f050006ea40 2024-03-25T19:41:33.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:33.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.882+0000 7f050570e640 10 monclient(hunting): authenticate will time out at 6009.168428s 2024-03-25T19:41:33.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.892+0000 7f04feffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f050006ea40 0x7f050013ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38656/0 (socket says 172.21.15.47:38656) 2024-03-25T19:41:33.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.893+0000 7f04feffd640 1 -- 172.21.15.47:0/1318665971 learned_addr learned my addr 172.21.15.47:0/1318665971 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:33.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.903+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307359809 0 0) 0x7f050006f1f0 con 0x7f0500071800 2024-03-25T19:41:33.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.903+0000 7f04fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:33.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.903+0000 7f04fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:33.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.903+0000 7f04fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:33.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.903+0000 7f04fdffb640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:41:33.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.903+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8004630 con 0x7f0500071800 2024-03-25T19:41:33.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.905+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390692884 0 0) 0x7f05000720b0 con 0x7f050006ea40 2024-03-25T19:41:33.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.905+0000 7f04fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:33.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.905+0000 7f04fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:33.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.905+0000 7f04fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:33.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.905+0000 7f04fdffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:41:33.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.905+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050006f1f0 con 0x7f050006ea40 2024-03-25T19:41:33.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 361192115 0 0) 0x7f04e8004630 con 0x7f0500071800 2024-03-25T19:41:33.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 >> v1:172.21.15.47:6790/0 conn(0x7f050006ea40 legacy=0x7f050013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 >> v1:172.21.15.142:6789/0 conn(0x7f0500071c40 legacy=0x7f050014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:33.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 1 monclient: found mon.noname-a 2024-03-25T19:41:33.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0500072780 con 0x7f0500071800 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f04f4001690 con 0x7f0500071800 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f050570e640 5 monclient: authenticate success, global_id 4539 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f050570e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f04fdffb640 10 monclient: dump: 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.912 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:33.913 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:33.913 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:33.913 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:33.913 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:33.913 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:33.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.911+0000 7f050570e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:33.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f40017f0 con 0x7f0500071800 2024-03-25T19:41:33.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f04fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:33.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f04fdffb640 1 -- 172.21.15.47:0/1318665971 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f04f4001c00 con 0x7f0500071800 2024-03-25T19:41:33.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f050570e640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f050570e640 4 set_mon_vals no callback set 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f04fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f04fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f04fdffb640 10 monclient: dump: 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:33.923 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.923+0000 7f050570e640 10 monclient: shutdown 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 1 -- 172.21.15.47:0/1318665971 >> v1:172.21.15.47:6789/0 conn(0x7f0500071800 legacy=0x7f0500147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 1 -- 172.21.15.47:0/1318665971 shutdown_connections 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 1 -- 172.21.15.47:0/1318665971 >> 172.21.15.47:0/1318665971 conn(0x7f050006b630 msgr2=0x7f05000b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 1 -- 172.21.15.47:0/1318665971 shutdown_connections 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 1 -- 172.21.15.47:0/1318665971 wait complete. 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 10 monclient: build_initial_monmap 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.924+0000 7f050570e640 10 monclient: monmap: 2024-03-25T19:41:33.925 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 1 librados: starting msgr at 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 1 librados: starting objecter 2024-03-25T19:41:33.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 1 Processor -- start 2024-03-25T19:41:33.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 1 -- start start 2024-03-25T19:41:33.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 1 librados: setting wanted keys 2024-03-25T19:41:33.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 1 librados: calling monclient init 2024-03-25T19:41:33.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.925+0000 7f050570e640 10 monclient: init 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient(hunting): picked mon.noname-b con 0x7f050006ea40 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient(hunting): picked mon.noname-c con 0x7f0500071800 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient(hunting): picked mon.noname-a con 0x7f0500071c40 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000aa180 con 0x7f0500071c40 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050011cb40 con 0x7f050006ea40 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050011cd70 con 0x7f0500071800 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:33.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.926+0000 7f050570e640 10 monclient(hunting): authenticate will time out at 6009.212763s 2024-03-25T19:41:33.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.935+0000 7f04ff7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0500071c40 0x7f05001d3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50708/0 (socket says 172.21.15.47:50708) 2024-03-25T19:41:33.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.935+0000 7f04ff7fe640 1 -- 172.21.15.47:0/2105316374 learned_addr learned my addr 172.21.15.47:0/2105316374 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:33.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.936+0000 7f04e77fe640 10 client.?.objecter ms_handle_connect 0x7f0500071c40 2024-03-25T19:41:33.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.936+0000 7f04e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:33.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.938+0000 7f04e77fe640 10 client.?.objecter ms_handle_connect 0x7f050006ea40 2024-03-25T19:41:33.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.938+0000 7f04e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:33.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364815056 0 0) 0x7f05000aa180 con 0x7f0500071c40 2024-03-25T19:41:33.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:33.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:33.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec004eb0 con 0x7f0500071c40 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139661129 0 0) 0x7f050011cb40 con 0x7f050006ea40 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:41:33.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.951+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05000aa180 con 0x7f050006ea40 2024-03-25T19:41:33.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.952+0000 7f04e77fe640 10 client.?.objecter ms_handle_connect 0x7f0500071800 2024-03-25T19:41:33.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.952+0000 7f04e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:33.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2288076565 0 0) 0x7f05000aa180 con 0x7f050006ea40 2024-03-25T19:41:33.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050011cb40 con 0x7f050006ea40 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f04f0001930 con 0x7f050006ea40 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 10 monclient(hunting): dump: 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:33.965 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:33.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.965+0000 7f04e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:33.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.980+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70337890 0 0) 0x7f050011cd70 con 0x7f0500071800 2024-03-25T19:41:33.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.980+0000 7f04e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:33.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.980+0000 7f04e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:33.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.980+0000 7f04e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:33.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.980+0000 7f04e77fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:41:33.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.980+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05000aa180 con 0x7f0500071800 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.983+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2725086210 0 0) 0x7f04ec004eb0 con 0x7f0500071c40 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.983+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050011cd70 con 0x7f0500071c40 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.983+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f04f8001c30 con 0x7f0500071c40 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.983+0000 7f04e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.983+0000 7f04e77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.983+0000 7f04e77fe640 10 monclient(hunting): dump: 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:33.984 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3940769624 0 0) 0x7f050011cb40 con 0x7f050006ea40 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 >> v1:172.21.15.47:6790/0 conn(0x7f0500071800 legacy=0x7f050013b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 >> v1:172.21.15.47:6789/0 conn(0x7f0500071c40 legacy=0x7f05001d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 1 monclient: found mon.b 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050013c540 con 0x7f050006ea40 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f04e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 10 monclient: _renew_subs 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:33.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05000ad3b0 con 0x7f050006ea40 2024-03-25T19:41:33.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:33.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 10 monclient: _renew_subs 2024-03-25T19:41:33.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:33.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f050013c140 con 0x7f050006ea40 2024-03-25T19:41:33.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.984+0000 7f050570e640 1 librados: init done 2024-03-25T19:41:33.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.985+0000 7f050570e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:33.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.985+0000 7f050570e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:33.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.985+0000 7f050570e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:33.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:33.985+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05000627b0 con 0x7f050006ea40 2024-03-25T19:41:34.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f0001d40 con 0x7f050006ea40 2024-03-25T19:41:34.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:34.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f04f0004a40 con 0x7f050006ea40 2024-03-25T19:41:34.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04e77fe640 10 monclient: dump: 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:34.261 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:34.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.260+0000 7f04fd7fa640 4 set_mon_vals no callback set 2024-03-25T19:41:34.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.261+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f04f0004c20 con 0x7f050006ea40 2024-03-25T19:41:34.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.262+0000 7f04fe7fc640 10 monclient: get_auth_request con 0x7f04ec06dfb0 auth_method 0 2024-03-25T19:41:34.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.262+0000 7f04e77fe640 10 client.4531.objecter ms_handle_connect 0x7f04ec06dfb0 2024-03-25T19:41:34.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f04f00547b0 con 0x7f050006ea40 2024-03-25T19:41:34.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 10 client.4531.objecter ms_dispatch 0x7f0500000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:34.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:34.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:34.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T19:41:34.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04f00500f0 con 0x7f050006ea40 2024-03-25T19:41:34.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:34.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.269+0000 7f04e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:34.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.507+0000 7f050570e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:34.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.507+0000 7f050570e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:34.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.507+0000 7f050570e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:34.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.507+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05000b0350 con 0x7f050006ea40 2024-03-25T19:41:34.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.533+0000 7f04e77fe640 1 -- 172.21.15.47:0/2105316374 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f04f00521f0 con 0x7f050006ea40 2024-03-25T19:41:34.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.533+0000 7f04e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:34.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.533+0000 7f04e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:34.533 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:34.533 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 10 librados: watch_flush enter 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 10 librados: watch_flush exit 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 10 client.4531.objecter successfully canceled tick 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 >> v1:172.21.15.142:6800/1025302268 conn(0x7f04ec06dfb0 legacy=0x7f04ec070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 10 monclient: shutdown 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 >> v1:172.21.15.142:6789/0 conn(0x7f050006ea40 legacy=0x7f05000acca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 shutdown_connections 2024-03-25T19:41:34.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 >> 172.21.15.47:0/2105316374 conn(0x7f050006b630 msgr2=0x7f05000ae370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:34.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 shutdown_connections 2024-03-25T19:41:34.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 -- 172.21.15.47:0/2105316374 wait complete. 2024-03-25T19:41:34.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:34.536+0000 7f050570e640 1 librados: shutdown 2024-03-25T19:41:34.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:41.355 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:41.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.462+0000 7f87b9225640 10 monclient: get_monmap_and_config 2024-03-25T19:41:41.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.462+0000 7f87b9225640 10 monclient: build_initial_monmap 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.462+0000 7f87b9225640 10 monclient: monmap: 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:41.463321+0000 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:41.463321+0000 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:41.462 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 1 Processor -- start 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 1 -- start start 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient: init 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient(hunting): picked mon.noname-c con 0x7f87b40b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient(hunting): picked mon.noname-a con 0x7f87b40b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient(hunting): picked mon.noname-b con 0x7f87b414be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40af940 con 0x7f87b40b1730 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40b0a70 con 0x7f87b40b4d60 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40b0d50 con 0x7f87b414be90 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:41.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.463+0000 7f87b9225640 10 monclient(hunting): authenticate will time out at 6016.749905s 2024-03-25T19:41:41.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.483+0000 7f87b2d9d640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f87b40b4d60 0x7f87b413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:35470/0 (socket says 172.21.15.47:35470) 2024-03-25T19:41:41.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.483+0000 7f87b2d9d640 1 -- 172.21.15.47:0/507635547 learned_addr learned my addr 172.21.15.47:0/507635547 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:41.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.514+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732064285 0 0) 0x7f87b40b0a70 con 0x7f87b40b4d60 2024-03-25T19:41:41.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.514+0000 7f87b259c640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.514+0000 7f87b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.514+0000 7f87b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.514+0000 7f87b259c640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:41:41.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.514+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a0003520 con 0x7f87b40b4d60 2024-03-25T19:41:41.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.523+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041885328 0 0) 0x7f87b40b0d50 con 0x7f87b414be90 2024-03-25T19:41:41.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.523+0000 7f87b259c640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.523+0000 7f87b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.523+0000 7f87b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.523+0000 7f87b259c640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:41:41.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.523+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b40b0a70 con 0x7f87b414be90 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065874300 0 0) 0x7f87a0003520 con 0x7f87b40b4d60 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 >> v1:172.21.15.47:6790/0 conn(0x7f87b414be90 legacy=0x7f87b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 >> v1:172.21.15.47:6789/0 conn(0x7f87b40b1730 legacy=0x7f87b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 10 monclient: _finish_hunting 0 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 1 monclient: found mon.noname-c 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b40b1030 con 0x7f87b40b4d60 2024-03-25T19:41:41.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 10 monclient: _finish_auth 0 2024-03-25T19:41:41.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 10 monclient: _check_auth_tickets 2024-03-25T19:41:41.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:41.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b9225640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:41:41.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.526+0000 7f87b9225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.540+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f87a4001650 con 0x7f87b40b4d60 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.540+0000 7f87b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.540+0000 7f87b259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.540+0000 7f87b259c640 10 monclient: dump: 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:41.540 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:41.541 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:41.541 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:41.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.540+0000 7f87b9225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:41.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87a40018a0 con 0x7f87b40b4d60 2024-03-25T19:41:41.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:41.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b259c640 1 -- 172.21.15.47:0/507635547 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f87a4001c30 con 0x7f87b40b4d60 2024-03-25T19:41:41.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b9225640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:41.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b9225640 4 set_mon_vals no callback set 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.550+0000 7f87b259c640 10 monclient: dump: 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:41.550 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:41.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 10 monclient: shutdown 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 1 -- 172.21.15.47:0/507635547 >> v1:172.21.15.142:6789/0 conn(0x7f87b40b4d60 legacy=0x7f87b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 1 -- 172.21.15.47:0/507635547 shutdown_connections 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 1 -- 172.21.15.47:0/507635547 >> 172.21.15.47:0/507635547 conn(0x7f87b40617a0 msgr2=0x7f87b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 1 -- 172.21.15.47:0/507635547 shutdown_connections 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.551+0000 7f87b9225640 1 -- 172.21.15.47:0/507635547 wait complete. 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 10 monclient: build_initial_monmap 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 10 monclient: monmap: 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:41.552 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 1 librados: starting msgr at 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 1 librados: starting objecter 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 1 Processor -- start 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 1 -- start start 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 1 librados: setting wanted keys 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 1 librados: calling monclient init 2024-03-25T19:41:41.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.552+0000 7f87b9225640 10 monclient: init 2024-03-25T19:41:41.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:41.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:41.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient(hunting): picked mon.noname-a con 0x7f87b40b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:41.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient(hunting): picked mon.noname-b con 0x7f87b40b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:41.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient(hunting): picked mon.noname-c con 0x7f87b414be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:41.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b4140a90 con 0x7f87b40b1730 2024-03-25T19:41:41.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b41d79d0 con 0x7f87b40b4d60 2024-03-25T19:41:41.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b4141ea0 con 0x7f87b414be90 2024-03-25T19:41:41.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:41.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.553+0000 7f87b9225640 10 monclient(hunting): authenticate will time out at 6016.839325s 2024-03-25T19:41:41.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.558+0000 7f87abfff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f87b40b4d60 0x7f87b41d7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:35512/0 (socket says 172.21.15.47:35512) 2024-03-25T19:41:41.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.558+0000 7f87abfff640 1 -- 172.21.15.47:0/3205191748 learned_addr learned my addr 172.21.15.47:0/3205191748 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:41.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.562+0000 7f87ab7fe640 10 client.?.objecter ms_handle_connect 0x7f87b40b4d60 2024-03-25T19:41:41.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.562+0000 7f87ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:41.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.565+0000 7f87ab7fe640 10 client.?.objecter ms_handle_connect 0x7f87b414be90 2024-03-25T19:41:41.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.565+0000 7f87ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:41.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.571+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277277240 0 0) 0x7f87b4141ea0 con 0x7f87b414be90 2024-03-25T19:41:41.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.571+0000 7f87ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.571+0000 7f87ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.571+0000 7f87ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.571+0000 7f87ab7fe640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:41:41.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.571+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8794004eb0 con 0x7f87b414be90 2024-03-25T19:41:41.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.580+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108068463 0 0) 0x7f87b41d79d0 con 0x7f87b40b4d60 2024-03-25T19:41:41.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.580+0000 7f87ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:41.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.580+0000 7f87ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:41.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.580+0000 7f87ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:41.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.580+0000 7f87ab7fe640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:41:41.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.580+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b4141ea0 con 0x7f87b40b4d60 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1419887891 0 0) 0x7f8794004eb0 con 0x7f87b414be90 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b41d79d0 con 0x7f87b414be90 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f87ac001bd0 con 0x7f87b414be90 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.581 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:41.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.581+0000 7f87ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:41.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.585+0000 7f87ab7fe640 10 client.?.objecter ms_handle_connect 0x7f87b40b1730 2024-03-25T19:41:41.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.585+0000 7f87ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:41.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1460825289 0 0) 0x7f87b41d79d0 con 0x7f87b414be90 2024-03-25T19:41:41.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 >> v1:172.21.15.142:6789/0 conn(0x7f87b40b4d60 legacy=0x7f87b41d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 >> v1:172.21.15.47:6789/0 conn(0x7f87b40b1730 legacy=0x7f87b4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:41.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:41.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 1 monclient: found mon.c 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b41d8570 con 0x7f87b414be90 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87b9225640 5 monclient: authenticate success, global_id 4529 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87b9225640 10 monclient: _renew_subs 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87b9225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.588+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b41d87e0 con 0x7f87b414be90 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 10 monclient: _renew_subs 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:41.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b4142780 con 0x7f87b414be90 2024-03-25T19:41:41.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 1 librados: init done 2024-03-25T19:41:41.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:41.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:41.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:41.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.589+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b40627b0 con 0x7f87b414be90 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ac002080 con 0x7f87b414be90 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f87ac003b30 con 0x7f87b414be90 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87ab7fe640 10 monclient: dump: 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:41.911 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.911+0000 7f87b1d9b640 4 set_mon_vals no callback set 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.912+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f87ac019b00 con 0x7f87b414be90 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.913+0000 7f87b2d9d640 10 monclient: get_auth_request con 0x7f879406dcc0 auth_method 0 2024-03-25T19:41:41.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.913+0000 7f87ab7fe640 10 client.4529.objecter ms_handle_connect 0x7f879406dcc0 2024-03-25T19:41:41.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f87ac053810 con 0x7f87b414be90 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 10 client.4529.objecter ms_dispatch 0x7f87b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87ac04f150 con 0x7f87b414be90 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:41.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:41.920+0000 7f87ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:42.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.177+0000 7f87b9225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:42.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.177+0000 7f87b9225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:42.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.177+0000 7f87b9225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:41:42.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.177+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87b4137970 con 0x7f87b414be90 2024-03-25T19:41:42.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.207+0000 7f87ab7fe640 1 -- 172.21.15.47:0/3205191748 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f87ac051250 con 0x7f87b414be90 2024-03-25T19:41:42.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.207+0000 7f87ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:42.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.207+0000 7f87ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:42.207 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:42.207 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.209+0000 7f87b9225640 10 librados: watch_flush enter 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 10 librados: watch_flush exit 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 10 client.4529.objecter successfully canceled tick 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 >> v1:172.21.15.142:6800/1025302268 conn(0x7f879406dcc0 legacy=0x7f8794070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 10 monclient: shutdown 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:42.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 >> v1:172.21.15.47:6790/0 conn(0x7f87b414be90 legacy=0x7f87b4142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 shutdown_connections 2024-03-25T19:41:42.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 >> 172.21.15.47:0/3205191748 conn(0x7f87b40617a0 msgr2=0x7f87b413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:42.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 shutdown_connections 2024-03-25T19:41:42.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 -- 172.21.15.47:0/3205191748 wait complete. 2024-03-25T19:41:42.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:42.210+0000 7f87b9225640 1 librados: shutdown 2024-03-25T19:41:42.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:43.298 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:20.533 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.343 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:07:20.533 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.344 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:07:20.534 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.354 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:20.589 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.368 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:07:20.558 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.384 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:20.620 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.421 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:20.656 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.428 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:20.664 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.456 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:20.692 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:49.027 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:49.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.142+0000 7fb0e587b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:49.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.142+0000 7fb0e587b640 10 monclient: build_initial_monmap 2024-03-25T19:41:49.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.142+0000 7fb0e587b640 10 monclient: monmap: 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:49.143723+0000 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:49.143723+0000 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:49.143 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.143+0000 7fb0e587b640 1 Processor -- start 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.143+0000 7fb0e587b640 1 -- start start 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.143+0000 7fb0e587b640 10 monclient: init 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e00a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e013f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e01430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e0142520 con 0x7fb0e00a98f0 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00b28a0 con 0x7fb0e01430c0 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00b2a30 con 0x7fb0e013f710 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:49.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.144+0000 7fb0e587b640 10 monclient(hunting): authenticate will time out at 6024.430726s 2024-03-25T19:41:49.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.158+0000 7fb0df7fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb0e01430c0 0x7fb0e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:34404/0 (socket says 172.21.15.47:34404) 2024-03-25T19:41:49.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.158+0000 7fb0df7fe640 1 -- 172.21.15.47:0/99675634 learned_addr learned my addr 172.21.15.47:0/99675634 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:49.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.173+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943764440 0 0) 0x7fb0e0142520 con 0x7fb0e00a98f0 2024-03-25T19:41:49.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.174+0000 7fb0ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.174+0000 7fb0ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.174+0000 7fb0ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.174+0000 7fb0ddffb640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:41:49.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.174+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c8003520 con 0x7fb0e00a98f0 2024-03-25T19:41:49.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.177+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031043695 0 0) 0x7fb0e00b28a0 con 0x7fb0e01430c0 2024-03-25T19:41:49.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.177+0000 7fb0ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.177+0000 7fb0ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.177+0000 7fb0ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.177+0000 7fb0ddffb640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:41:49.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.177+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0142520 con 0x7fb0e01430c0 2024-03-25T19:41:49.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.191+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1880067193 0 0) 0x7fb0e0142520 con 0x7fb0e01430c0 2024-03-25T19:41:49.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 >> v1:172.21.15.47:6790/0 conn(0x7fb0e013f710 legacy=0x7fb0e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 >> v1:172.21.15.47:6789/0 conn(0x7fb0e00a98f0 legacy=0x7fb0e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 1 monclient: found mon.noname-c 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e00b44c0 con 0x7fb0e01430c0 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb0d80019f0 con 0x7fb0e01430c0 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0ddffb640 10 monclient: dump: 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.192 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0e587b640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:41:49.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.192+0000 7fb0e587b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d8001b50 con 0x7fb0e01430c0 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0ddffb640 1 -- 172.21.15.47:0/99675634 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb0d8001f20 con 0x7fb0e01430c0 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0e587b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0e587b640 4 set_mon_vals no callback set 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.209+0000 7fb0ddffb640 10 monclient: dump: 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.209 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.210 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.210 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:49.210 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:49.210 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:49.210 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 10 monclient: shutdown 2024-03-25T19:41:49.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 1 -- 172.21.15.47:0/99675634 >> v1:172.21.15.142:6789/0 conn(0x7fb0e01430c0 legacy=0x7fb0e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 1 -- 172.21.15.47:0/99675634 shutdown_connections 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 1 -- 172.21.15.47:0/99675634 >> 172.21.15.47:0/99675634 conn(0x7fb0e00b7ea0 msgr2=0x7fb0e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 1 -- 172.21.15.47:0/99675634 shutdown_connections 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.210+0000 7fb0e587b640 1 -- 172.21.15.47:0/99675634 wait complete. 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.211+0000 7fb0e587b640 10 monclient: build_initial_monmap 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.211+0000 7fb0e587b640 10 monclient: monmap: 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:49.212 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 1 librados: starting msgr at 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 1 librados: starting objecter 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 1 Processor -- start 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 1 -- start start 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 1 librados: setting wanted keys 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 1 librados: calling monclient init 2024-03-25T19:41:49.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.212+0000 7fb0e587b640 10 monclient: init 2024-03-25T19:41:49.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:49.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:49.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e00a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e01430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e0138600 con 0x7fb0e00a98f0 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00b1620 con 0x7fb0e013f710 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e01346b0 con 0x7fb0e01430c0 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:49.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.213+0000 7fb0e587b640 10 monclient(hunting): authenticate will time out at 6024.499458s 2024-03-25T19:41:49.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.229+0000 7fb0deffd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb0e00a98f0 0x7fb0e0137690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35146/0 (socket says 172.21.15.47:35146) 2024-03-25T19:41:49.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.229+0000 7fb0deffd640 1 -- 172.21.15.47:0/21329413 learned_addr learned my addr 172.21.15.47:0/21329413 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:49.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.232+0000 7fb0c7fff640 10 client.?.objecter ms_handle_connect 0x7fb0e00a98f0 2024-03-25T19:41:49.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.232+0000 7fb0c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:49.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.233+0000 7fb0c7fff640 10 client.?.objecter ms_handle_connect 0x7fb0e013f710 2024-03-25T19:41:49.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.233+0000 7fb0c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.234+0000 7fb0c7fff640 10 client.?.objecter ms_handle_connect 0x7fb0e01430c0 2024-03-25T19:41:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.234+0000 7fb0c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.240+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743936870 0 0) 0x7fb0e00b1620 con 0x7fb0e013f710 2024-03-25T19:41:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.240+0000 7fb0c7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.240+0000 7fb0c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.240+0000 7fb0c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.240+0000 7fb0c7fff640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:41:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.240+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc004eb0 con 0x7fb0e013f710 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814328196 0 0) 0x7fb0e0138600 con 0x7fb0e00a98f0 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e00b1620 con 0x7fb0e00a98f0 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.241+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120387660 0 0) 0x7fb0e01346b0 con 0x7fb0e01430c0 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.242+0000 7fb0c7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.242+0000 7fb0c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.242+0000 7fb0c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:49.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.242+0000 7fb0c7fff640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:41:49.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.242+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0138600 con 0x7fb0e01430c0 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438489722 0 0) 0x7fb0cc004eb0 con 0x7fb0e013f710 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e01346b0 con 0x7fb0e013f710 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb0d40018c0 con 0x7fb0e013f710 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 10 monclient(hunting): dump: 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.248 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.248+0000 7fb0c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.253+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 920282632 0 0) 0x7fb0e0138600 con 0x7fb0e01430c0 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.253+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc004eb0 con 0x7fb0e01430c0 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.253+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb0d8001e60 con 0x7fb0e01430c0 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.253+0000 7fb0c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.253+0000 7fb0c7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.253+0000 7fb0c7fff640 10 monclient(hunting): dump: 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:49.253 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:49.254 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842415431 0 0) 0x7fb0e00b1620 con 0x7fb0e00a98f0 2024-03-25T19:41:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e0138600 con 0x7fb0e00a98f0 2024-03-25T19:41:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb0d0001990 con 0x7fb0e00a98f0 2024-03-25T19:41:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient(hunting): dump: 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1690315832 0 0) 0x7fb0e01346b0 con 0x7fb0e013f710 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 >> v1:172.21.15.47:6790/0 conn(0x7fb0e01430c0 legacy=0x7fb0e01da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 >> v1:172.21.15.47:6789/0 conn(0x7fb0e00a98f0 legacy=0x7fb0e0137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 monclient: found mon.b 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0134840 con 0x7fb0e013f710 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0e587b640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0e587b640 10 monclient: _renew_subs 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0e587b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.257+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e0137f50 con 0x7fb0e013f710 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.258+0000 7fb0e587b640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.258+0000 7fb0e587b640 10 monclient: _renew_subs 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.258+0000 7fb0e587b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:49.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.258+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e01daf90 con 0x7fb0e013f710 2024-03-25T19:41:49.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.258+0000 7fb0e587b640 1 librados: init done 2024-03-25T19:41:49.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.259+0000 7fb0e587b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:49.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.259+0000 7fb0e587b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:49.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.259+0000 7fb0e587b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:49.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.259+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e00627b0 con 0x7fb0e013f710 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d40035f0 con 0x7fb0e013f710 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb0d4003900 con 0x7fb0e013f710 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0c7fff640 10 monclient: dump: 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:49.546 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:49.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.546+0000 7fb0dd7fa640 4 set_mon_vals no callback set 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.547+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fb0d40198d0 con 0x7fb0e013f710 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.547+0000 7fb0deffd640 10 monclient: get_auth_request con 0x7fb0cc06e0b0 auth_method 0 2024-03-25T19:41:49.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.548+0000 7fb0c7fff640 10 client.4543.objecter ms_handle_connect 0x7fb0cc06e0b0 2024-03-25T19:41:49.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.552+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fb0d40535f0 con 0x7fb0e013f710 2024-03-25T19:41:49.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.552+0000 7fb0c7fff640 10 client.4543.objecter ms_dispatch 0x7fb0e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:49.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.552+0000 7fb0c7fff640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:49.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.552+0000 7fb0c7fff640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:49.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.552+0000 7fb0c7fff640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.553+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0d404ef30 con 0x7fb0e013f710 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.553+0000 7fb0c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:49.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.553+0000 7fb0c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:49.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.777+0000 7fb0e587b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:49.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.777+0000 7fb0e587b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:49.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.777+0000 7fb0e587b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:49.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.777+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0e0138790 con 0x7fb0e013f710 2024-03-25T19:41:49.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.815+0000 7fb0c7fff640 1 -- 172.21.15.47:0/21329413 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fb0d4051030 con 0x7fb0e013f710 2024-03-25T19:41:49.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.815+0000 7fb0c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:49.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.815+0000 7fb0c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:49.815 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:49.815 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 10 librados: watch_flush enter 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 10 librados: watch_flush exit 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 10 client.4543.objecter successfully canceled tick 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 >> v1:172.21.15.142:6800/1025302268 conn(0x7fb0cc06e0b0 legacy=0x7fb0cc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 10 monclient: shutdown 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 >> v1:172.21.15.142:6789/0 conn(0x7fb0e013f710 legacy=0x7fb0e0133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 shutdown_connections 2024-03-25T19:41:49.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 >> 172.21.15.47:0/21329413 conn(0x7fb0e00b7ea0 msgr2=0x7fb0e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:49.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 shutdown_connections 2024-03-25T19:41:49.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 -- 172.21.15.47:0/21329413 wait complete. 2024-03-25T19:41:49.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:49.818+0000 7fb0e587b640 1 librados: shutdown 2024-03-25T19:41:49.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:56.639 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.741+0000 7f49c71f4640 10 monclient: get_monmap_and_config 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.741+0000 7f49c71f4640 10 monclient: build_initial_monmap 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.741+0000 7f49c71f4640 10 monclient: monmap: 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:41:56.742221+0000 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:41:56.742221+0000 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:41:56.741 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:41:56.742 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.742+0000 7f49c71f4640 1 Processor -- start 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.742+0000 7f49c71f4640 1 -- start start 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.742+0000 7f49c71f4640 10 monclient: init 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient(hunting): picked mon.noname-a con 0x7f49c00b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:56.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient(hunting): picked mon.noname-c con 0x7f49c00b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:56.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient(hunting): picked mon.noname-b con 0x7f49c014be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:56.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00af940 con 0x7f49c00b4d60 2024-03-25T19:41:56.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00b0a70 con 0x7f49c00b1730 2024-03-25T19:41:56.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00b0d50 con 0x7f49c014be90 2024-03-25T19:41:56.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:56.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.743+0000 7f49c71f4640 10 monclient(hunting): authenticate will time out at 6032.029145s 2024-03-25T19:41:56.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.752+0000 7f49bffff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f49c00b1730 0x7f49c00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:39130/0 (socket says 172.21.15.47:39130) 2024-03-25T19:41:56.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.753+0000 7f49bffff640 1 -- 172.21.15.47:0/1687669261 learned_addr learned my addr 172.21.15.47:0/1687669261 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:56.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.765+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16601701 0 0) 0x7f49c00b0d50 con 0x7f49c014be90 2024-03-25T19:41:56.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.765+0000 7f49bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:56.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.765+0000 7f49bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:56.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.765+0000 7f49bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:56.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.765+0000 7f49bf7fe640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:41:56.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.765+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b0003520 con 0x7f49c014be90 2024-03-25T19:41:56.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.773+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850662336 0 0) 0x7f49c00af940 con 0x7f49c00b4d60 2024-03-25T19:41:56.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.773+0000 7f49bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:56.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.773+0000 7f49bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:56.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.773+0000 7f49bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:56.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.773+0000 7f49bf7fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:41:56.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.773+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c00b0d50 con 0x7f49c00b4d60 2024-03-25T19:41:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.774+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436368453 0 0) 0x7f49c00b0a70 con 0x7f49c00b1730 2024-03-25T19:41:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.774+0000 7f49bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.774+0000 7f49bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.774+0000 7f49bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.774+0000 7f49bf7fe640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:41:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.774+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c00af940 con 0x7f49c00b1730 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419977665 0 0) 0x7f49b0003520 con 0x7f49c014be90 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 >> v1:172.21.15.142:6789/0 conn(0x7f49c00b1730 legacy=0x7f49c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 >> v1:172.21.15.47:6789/0 conn(0x7f49c00b4d60 legacy=0x7f49c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 1 monclient: found mon.noname-b 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00b1030 con 0x7f49c014be90 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f49b8001a10 con 0x7f49c014be90 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:56.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49bf7fe640 10 monclient: dump: 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49c71f4640 5 monclient: authenticate success, global_id 4538 2024-03-25T19:41:56.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.785+0000 7f49c71f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b8001b70 con 0x7f49c014be90 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49bf7fe640 1 -- 172.21.15.47:0/1687669261 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f49b8001f80 con 0x7f49c014be90 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49bf7fe640 10 monclient: dump: 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:56.798 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:56.799 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:56.799 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:56.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49c71f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:56.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.798+0000 7f49c71f4640 4 set_mon_vals no callback set 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.799+0000 7f49c71f4640 10 monclient: shutdown 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.799+0000 7f49c71f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.799+0000 7f49c71f4640 1 -- 172.21.15.47:0/1687669261 >> v1:172.21.15.47:6790/0 conn(0x7f49c014be90 legacy=0x7f49c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.799+0000 7f49c71f4640 1 -- 172.21.15.47:0/1687669261 shutdown_connections 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.799+0000 7f49c71f4640 1 -- 172.21.15.47:0/1687669261 >> 172.21.15.47:0/1687669261 conn(0x7f49c00617a0 msgr2=0x7f49c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.800+0000 7f49c71f4640 1 -- 172.21.15.47:0/1687669261 shutdown_connections 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.800+0000 7f49c71f4640 1 -- 172.21.15.47:0/1687669261 wait complete. 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 10 monclient: build_initial_monmap 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 10 monclient: monmap: 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:56.800 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:41:56.801 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:56.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 1 librados: starting msgr at 2024-03-25T19:41:56.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 1 librados: starting objecter 2024-03-25T19:41:56.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 1 Processor -- start 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 1 -- start start 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 1 librados: setting wanted keys 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 1 librados: calling monclient init 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.801+0000 7f49c71f4640 10 monclient: init 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient(hunting): picked mon.noname-b con 0x7f49c00b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient(hunting): picked mon.noname-a con 0x7f49c00b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient(hunting): picked mon.noname-c con 0x7f49c014be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c0137220 con 0x7f49c00b4d60 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c0145330 con 0x7f49c00b1730 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c013e750 con 0x7f49c014be90 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:56.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.802+0000 7f49c71f4640 10 monclient(hunting): authenticate will time out at 6032.088512s 2024-03-25T19:41:56.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.822+0000 7f49c4f90640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f49c00b1730 0x7f49c01dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:39164/0 (socket says 172.21.15.47:39164) 2024-03-25T19:41:56.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.822+0000 7f49c4f90640 1 -- 172.21.15.47:0/3113298660 learned_addr learned my addr 172.21.15.47:0/3113298660 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:41:56.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.822+0000 7f49bd7fa640 10 client.?.objecter ms_handle_connect 0x7f49c00b1730 2024-03-25T19:41:56.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.822+0000 7f49bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:56.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.833+0000 7f49bd7fa640 10 client.?.objecter ms_handle_connect 0x7f49c00b4d60 2024-03-25T19:41:56.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.833+0000 7f49bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:56.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.837+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658621855 0 0) 0x7f49c0145330 con 0x7f49c00b1730 2024-03-25T19:41:56.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.837+0000 7f49bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:56.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.837+0000 7f49bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:56.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.837+0000 7f49bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:56.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.837+0000 7f49bd7fa640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:41:56.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.837+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a4004eb0 con 0x7f49c00b1730 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.844+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004523636 0 0) 0x7f49c0137220 con 0x7f49c00b4d60 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.844+0000 7f49bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.844+0000 7f49bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.844+0000 7f49bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.844+0000 7f49bd7fa640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.844+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c0145330 con 0x7f49c00b4d60 2024-03-25T19:41:56.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.845+0000 7f49bd7fa640 10 client.?.objecter ms_handle_connect 0x7f49c014be90 2024-03-25T19:41:56.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.845+0000 7f49bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865742645 0 0) 0x7f49a4004eb0 con 0x7f49c00b1730 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49c0137220 con 0x7f49c00b1730 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f49b4001a30 con 0x7f49c00b1730 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 10 monclient(hunting): dump: 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:56.860 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.860+0000 7f49bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.862+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249429330 0 0) 0x7f49c013e750 con 0x7f49c014be90 2024-03-25T19:41:56.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.862+0000 7f49bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:56.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.862+0000 7f49bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:56.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.862+0000 7f49bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:56.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.862+0000 7f49bd7fa640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:41:56.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.862+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a4004eb0 con 0x7f49c014be90 2024-03-25T19:41:56.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 640675780 0 0) 0x7f49c0137220 con 0x7f49c00b1730 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 >> v1:172.21.15.47:6790/0 conn(0x7f49c014be90 legacy=0x7f49c0144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 >> v1:172.21.15.47:6789/0 conn(0x7f49c00b4d60 legacy=0x7f49c0136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 1 monclient: found mon.b 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c01dba10 con 0x7f49c00b1730 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 10 monclient: _renew_subs 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c013dba0 con 0x7f49c00b1730 2024-03-25T19:41:56.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 10 monclient: _renew_subs 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c013ee50 con 0x7f49c00b1730 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.885+0000 7f49c71f4640 1 librados: init done 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.886+0000 7f49c71f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.886+0000 7f49c71f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:56.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.886+0000 7f49c71f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:56.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:56.886+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c013f100 con 0x7f49c00b1730 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b4001c70 con 0x7f49c00b1730 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f49b4003a60 con 0x7f49c00b1730 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49bd7fa640 10 monclient: dump: 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.199 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.199+0000 7f49beffd640 4 set_mon_vals no callback set 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.200+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f49b4003c80 con 0x7f49c00b1730 2024-03-25T19:41:57.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.200+0000 7f49bffff640 10 monclient: get_auth_request con 0x7f49a406de00 auth_method 0 2024-03-25T19:41:57.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.201+0000 7f49bd7fa640 10 client.4549.objecter ms_handle_connect 0x7f49a406de00 2024-03-25T19:41:57.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f49b40548d0 con 0x7f49c00b1730 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 10 client.4549.objecter ms_dispatch 0x7f49c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49b40503e0 con 0x7f49c00b1730 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:57.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.205+0000 7f49bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:57.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.458+0000 7f49c71f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:57.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.458+0000 7f49c71f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:57.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.458+0000 7f49c71f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:41:57.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.458+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49c013e8e0 con 0x7f49c00b1730 2024-03-25T19:41:57.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.492+0000 7f49bd7fa640 1 -- 172.21.15.47:0/3113298660 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f49b4052310 con 0x7f49c00b1730 2024-03-25T19:41:57.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.492+0000 7f49bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:57.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.492+0000 7f49bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:57.492 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:41:57.492 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:57.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 10 librados: watch_flush enter 2024-03-25T19:41:57.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 10 librados: watch_flush exit 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 10 client.4549.objecter successfully canceled tick 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 >> v1:172.21.15.142:6800/1025302268 conn(0x7f49a406de00 legacy=0x7f49a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 10 monclient: shutdown 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.495+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 >> v1:172.21.15.142:6789/0 conn(0x7f49c00b1730 legacy=0x7f49c01dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.496+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 shutdown_connections 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.496+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 >> 172.21.15.47:0/3113298660 conn(0x7f49c00617a0 msgr2=0x7f49c00b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.496+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 shutdown_connections 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.496+0000 7f49c71f4640 1 -- 172.21.15.47:0/3113298660 wait complete. 2024-03-25T19:41:57.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:41:57.496+0000 7f49c71f4640 1 librados: shutdown 2024-03-25T19:41:57.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:03.428 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:40.663 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.432 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:40.668 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.464 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:40.700 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.484 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:07:40.674 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.486 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:40.722 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.492 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:07:40.682 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.496 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:40.732 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.513 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:07:40.703 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:03.518 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:40.754 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:04.313 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.428+0000 7f73c9649640 10 monclient: get_monmap_and_config 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.428+0000 7f73c9649640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.428+0000 7f73c9649640 10 monclient: monmap: 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:04.430455+0000 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:04.430455+0000 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:04.429 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:04.430 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:04.430 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:04.430 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:04.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.429+0000 7f73c9649640 1 Processor -- start 2024-03-25T19:42:04.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.429+0000 7f73c9649640 1 -- start start 2024-03-25T19:42:04.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.429+0000 7f73c9649640 10 monclient: init 2024-03-25T19:42:04.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient(hunting): picked mon.noname-b con 0x7f73c40a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient(hunting): picked mon.noname-a con 0x7f73c413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient(hunting): picked mon.noname-c con 0x7f73c41430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c4142520 con 0x7f73c413f710 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40b28a0 con 0x7f73c41430c0 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40b2a30 con 0x7f73c40a98f0 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.430+0000 7f73c9649640 10 monclient(hunting): authenticate will time out at 6039.717092s 2024-03-25T19:42:04.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.436+0000 7f73c2ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f73c40a98f0 0x7f73c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47944/0 (socket says 172.21.15.47:47944) 2024-03-25T19:42:04.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.436+0000 7f73c2ffd640 1 -- 172.21.15.47:0/3586066643 learned_addr learned my addr 172.21.15.47:0/3586066643 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:04.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.446+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428415417 0 0) 0x7f73c40b2a30 con 0x7f73c40a98f0 2024-03-25T19:42:04.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.446+0000 7f73c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.446+0000 7f73c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.446+0000 7f73c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.446+0000 7f73c27fc640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:42:04.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.446+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b0003520 con 0x7f73c40a98f0 2024-03-25T19:42:04.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.463+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609945124 0 0) 0x7f73c40b28a0 con 0x7f73c41430c0 2024-03-25T19:42:04.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.463+0000 7f73c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.463+0000 7f73c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.463+0000 7f73c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.463+0000 7f73c27fc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:42:04.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.463+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c40b2a30 con 0x7f73c41430c0 2024-03-25T19:42:04.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3307150661 0 0) 0x7f73b0003520 con 0x7f73c40a98f0 2024-03-25T19:42:04.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 >> v1:172.21.15.142:6789/0 conn(0x7f73c41430c0 legacy=0x7f73c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 >> v1:172.21.15.47:6789/0 conn(0x7f73c413f710 legacy=0x7f73c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 1 monclient: found mon.noname-b 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c40b44c0 con 0x7f73c40a98f0 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f73b4001690 con 0x7f73c40a98f0 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c9649640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c9649640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c27fc640 10 monclient: dump: 2024-03-25T19:42:04.474 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:04.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.472+0000 7f73c9649640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.478+0000 7f73c27fc640 1 -- 172.21.15.47:0/3586066643 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73b40017f0 con 0x7f73c40a98f0 2024-03-25T19:42:04.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.478+0000 7f73c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.478+0000 7f73c9649640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:04.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.478+0000 7f73c9649640 4 set_mon_vals no callback set 2024-03-25T19:42:04.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 10 monclient: shutdown 2024-03-25T19:42:04.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 1 -- 172.21.15.47:0/3586066643 >> v1:172.21.15.47:6790/0 conn(0x7f73c40a98f0 legacy=0x7f73c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 1 -- 172.21.15.47:0/3586066643 shutdown_connections 2024-03-25T19:42:04.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 1 -- 172.21.15.47:0/3586066643 >> 172.21.15.47:0/3586066643 conn(0x7f73c40b7ea0 msgr2=0x7f73c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 1 -- 172.21.15.47:0/3586066643 shutdown_connections 2024-03-25T19:42:04.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.479+0000 7f73c9649640 1 -- 172.21.15.47:0/3586066643 wait complete. 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 10 monclient: monmap: 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:04.481 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 1 librados: starting msgr at 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 1 librados: starting objecter 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 1 Processor -- start 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 1 -- start start 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 1 librados: setting wanted keys 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 1 librados: calling monclient init 2024-03-25T19:42:04.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.480+0000 7f73c9649640 10 monclient: init 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient(hunting): picked mon.noname-c con 0x7f73c40a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient(hunting): picked mon.noname-b con 0x7f73c413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient(hunting): picked mon.noname-a con 0x7f73c41430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c413cc30 con 0x7f73c41430c0 2024-03-25T19:42:04.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40b71e0 con 0x7f73c413f710 2024-03-25T19:42:04.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c413afc0 con 0x7f73c40a98f0 2024-03-25T19:42:04.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.481+0000 7f73c9649640 10 monclient(hunting): authenticate will time out at 6039.768363s 2024-03-25T19:42:04.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.489+0000 7f73c37fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f73c41430c0 0x7f73c41e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38036/0 (socket says 172.21.15.47:38036) 2024-03-25T19:42:04.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.489+0000 7f73c37fe640 1 -- 172.21.15.47:0/1852841096 learned_addr learned my addr 172.21.15.47:0/1852841096 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:04.490 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.490+0000 7f73bb7fe640 10 client.?.objecter ms_handle_connect 0x7f73c41430c0 2024-03-25T19:42:04.490 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.490+0000 7f73bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.495+0000 7f73bb7fe640 10 client.?.objecter ms_handle_connect 0x7f73c413f710 2024-03-25T19:42:04.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.495+0000 7f73bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.495+0000 7f73bb7fe640 10 client.?.objecter ms_handle_connect 0x7f73c40a98f0 2024-03-25T19:42:04.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.495+0000 7f73bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.501+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379747486 0 0) 0x7f73c413afc0 con 0x7f73c40a98f0 2024-03-25T19:42:04.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.501+0000 7f73bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.501+0000 7f73bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.501+0000 7f73bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.501+0000 7f73bb7fe640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:42:04.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.501+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a4004eb0 con 0x7f73c40a98f0 2024-03-25T19:42:04.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.502+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176887894 0 0) 0x7f73c40b71e0 con 0x7f73c413f710 2024-03-25T19:42:04.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.502+0000 7f73bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.502+0000 7f73bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.502+0000 7f73bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.502+0000 7f73bb7fe640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:42:04.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.502+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c413afc0 con 0x7f73c413f710 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114690037 0 0) 0x7f73a4004eb0 con 0x7f73c40a98f0 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73c40b71e0 con 0x7f73c40a98f0 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f73b4001ca0 con 0x7f73c40a98f0 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.514 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:04.515 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:04.515 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:04.515 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:04.515 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:04.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.513+0000 7f73bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:04.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.517+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727150037 0 0) 0x7f73c413cc30 con 0x7f73c41430c0 2024-03-25T19:42:04.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.517+0000 7f73bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.517+0000 7f73bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.517+0000 7f73bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.517+0000 7f73bb7fe640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:42:04.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.517+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a4004eb0 con 0x7f73c41430c0 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1889780544 0 0) 0x7f73c40b71e0 con 0x7f73c40a98f0 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 >> v1:172.21.15.142:6789/0 conn(0x7f73c413f710 legacy=0x7f73c4139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 >> v1:172.21.15.47:6789/0 conn(0x7f73c41430c0 legacy=0x7f73c41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 1 monclient: found mon.c 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c413b2a0 con 0x7f73c40a98f0 2024-03-25T19:42:04.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 10 monclient: _renew_subs 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c413b580 con 0x7f73c40a98f0 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 10 monclient: _renew_subs 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c40b7ca0 con 0x7f73c40a98f0 2024-03-25T19:42:04.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.525+0000 7f73c9649640 1 librados: init done 2024-03-25T19:42:04.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.526+0000 7f73c9649640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.526+0000 7f73c9649640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.526+0000 7f73c9649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:04.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.526+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c40627b0 con 0x7f73c40a98f0 2024-03-25T19:42:04.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73b4004a90 con 0x7f73c40a98f0 2024-03-25T19:42:04.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f73b4004da0 con 0x7f73c40a98f0 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73bb7fe640 10 monclient: dump: 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:04.836 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:04.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.835+0000 7f73c1ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:04.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.836+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f73b401ad70 con 0x7f73c40a98f0 2024-03-25T19:42:04.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.837+0000 7f73bbfff640 10 monclient: get_auth_request con 0x7f73a406de00 auth_method 0 2024-03-25T19:42:04.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.837+0000 7f73bb7fe640 10 client.4547.objecter ms_handle_connect 0x7f73a406de00 2024-03-25T19:42:04.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f73b40548a0 con 0x7f73c40a98f0 2024-03-25T19:42:04.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 10 client.4547.objecter ms_dispatch 0x7f73c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:04.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:04.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:04.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T19:42:04.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73b40501e0 con 0x7f73c40a98f0 2024-03-25T19:42:04.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:04.844+0000 7f73bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:05.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.070+0000 7f73c9649640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.070+0000 7f73c9649640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.070+0000 7f73c9649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:05.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.070+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c413f0a0 con 0x7f73c40a98f0 2024-03-25T19:42:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.121+0000 7f73bb7fe640 1 -- 172.21.15.47:0/1852841096 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f73b40522e0 con 0x7f73c40a98f0 2024-03-25T19:42:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.121+0000 7f73bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.121+0000 7f73bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:05.122 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:05.122 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:05.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 10 librados: watch_flush enter 2024-03-25T19:42:05.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 10 librados: watch_flush exit 2024-03-25T19:42:05.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 10 client.4547.objecter successfully canceled tick 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 >> v1:172.21.15.142:6800/1025302268 conn(0x7f73a406de00 legacy=0x7f73a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 10 monclient: shutdown 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 >> v1:172.21.15.47:6790/0 conn(0x7f73c40a98f0 legacy=0x7f73c413b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 shutdown_connections 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.123+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 >> 172.21.15.47:0/1852841096 conn(0x7f73c40b7ea0 msgr2=0x7f73c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.124+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 shutdown_connections 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.124+0000 7f73c9649640 1 -- 172.21.15.47:0/1852841096 wait complete. 2024-03-25T19:42:05.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:05.124+0000 7f73c9649640 1 librados: shutdown 2024-03-25T19:42:05.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:11.943 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:12.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.058+0000 7f9f4472b640 10 monclient: get_monmap_and_config 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.058+0000 7f9f4472b640 10 monclient: build_initial_monmap 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.058+0000 7f9f4472b640 10 monclient: monmap: 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:12.060244+0000 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:12.060244+0000 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:12.059 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.059+0000 7f9f4472b640 1 Processor -- start 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.059+0000 7f9f4472b640 1 -- start start 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.059+0000 7f9f4472b640 10 monclient: init 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3c13fc60 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3c070570 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3c0727a0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c13c9f0 con 0x7f9f3c070570 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c06fac0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0715d0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:12.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.060+0000 7f9f4472b640 10 monclient(hunting): authenticate will time out at 6047.347075s 2024-03-25T19:42:12.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.070+0000 7f9f424c7640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9f3c13fc60 0x7f9f3c06f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:52962/0 (socket says 172.21.15.47:52962) 2024-03-25T19:42:12.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.070+0000 7f9f424c7640 1 -- 172.21.15.47:0/1499888719 learned_addr learned my addr 172.21.15.47:0/1499888719 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:12.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.086+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029010115 0 0) 0x7f9f3c06fac0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.086+0000 7f9f414c5640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.086+0000 7f9f414c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.086+0000 7f9f414c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.086+0000 7f9f414c5640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:42:12.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.086+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f28003520 con 0x7f9f3c13fc60 2024-03-25T19:42:12.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.093+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859798079 0 0) 0x7f9f3c13c9f0 con 0x7f9f3c070570 2024-03-25T19:42:12.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.093+0000 7f9f414c5640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.093+0000 7f9f414c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.093+0000 7f9f414c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.093+0000 7f9f414c5640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:42:12.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.093+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3c06fac0 con 0x7f9f3c070570 2024-03-25T19:42:12.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.099+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257807960 0 0) 0x7f9f3c0715d0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.099+0000 7f9f414c5640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.099+0000 7f9f414c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.099+0000 7f9f414c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.099+0000 7f9f414c5640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:42:12.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.099+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3c13c9f0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023544802 0 0) 0x7f9f28003520 con 0x7f9f3c13fc60 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 >> v1:172.21.15.47:6790/0 conn(0x7f9f3c0727a0 legacy=0x7f9f3c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 >> v1:172.21.15.47:6789/0 conn(0x7f9f3c070570 legacy=0x7f9f3c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: _finish_hunting 0 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 1 monclient: found mon.noname-c 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0718b0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: _finish_auth 0 2024-03-25T19:42:12.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: _check_auth_tickets 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9f38001690 con 0x7f9f3c13fc60 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f4472b640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f4472b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f414c5640 10 monclient: dump: 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:12.104 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:12.105 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:12.105 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:12.105 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.102+0000 7f9f4472b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:12.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f380017f0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f414c5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f414c5640 1 -- 172.21.15.47:0/1499888719 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9f38001c00 con 0x7f9f3c13fc60 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f414c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f414c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f414c5640 10 monclient: dump: 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:12.111 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f4472b640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:12.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.110+0000 7f9f4472b640 4 set_mon_vals no callback set 2024-03-25T19:42:12.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 10 monclient: shutdown 2024-03-25T19:42:12.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:12.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 1 -- 172.21.15.47:0/1499888719 >> v1:172.21.15.142:6789/0 conn(0x7f9f3c13fc60 legacy=0x7f9f3c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.112 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 1 -- 172.21.15.47:0/1499888719 shutdown_connections 2024-03-25T19:42:12.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 1 -- 172.21.15.47:0/1499888719 >> 172.21.15.47:0/1499888719 conn(0x7f9f3c0b4a80 msgr2=0x7f9f3c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:12.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 1 -- 172.21.15.47:0/1499888719 shutdown_connections 2024-03-25T19:42:12.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.111+0000 7f9f4472b640 1 -- 172.21.15.47:0/1499888719 wait complete. 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.112+0000 7f9f4472b640 10 monclient: build_initial_monmap 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.112+0000 7f9f4472b640 10 monclient: monmap: 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.114 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 1 librados: starting msgr at 2024-03-25T19:42:12.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 1 librados: starting objecter 2024-03-25T19:42:12.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 1 Processor -- start 2024-03-25T19:42:12.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 1 -- start start 2024-03-25T19:42:12.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 1 librados: setting wanted keys 2024-03-25T19:42:12.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 1 librados: calling monclient init 2024-03-25T19:42:12.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.113+0000 7f9f4472b640 10 monclient: init 2024-03-25T19:42:12.116 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:12.116 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:12.116 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3c070570 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:12.116 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3c0727a0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:12.116 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3c13fc60 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:12.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c138340 con 0x7f9f3c070570 2024-03-25T19:42:12.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c138020 con 0x7f9f3c0727a0 2024-03-25T19:42:12.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0ae1e0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:12.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.114+0000 7f9f4472b640 10 monclient(hunting): authenticate will time out at 6047.401529s 2024-03-25T19:42:12.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.123+0000 7f9f41cc6640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9f3c0727a0 0x7f9f3c1d79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:52986/0 (socket says 172.21.15.47:52986) 2024-03-25T19:42:12.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.123+0000 7f9f41cc6640 1 -- 172.21.15.47:0/108885897 learned_addr learned my addr 172.21.15.47:0/108885897 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:12.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.127+0000 7f9f32ffd640 10 client.?.objecter ms_handle_connect 0x7f9f3c0727a0 2024-03-25T19:42:12.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.127+0000 7f9f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.136+0000 7f9f32ffd640 10 client.?.objecter ms_handle_connect 0x7f9f3c070570 2024-03-25T19:42:12.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.136+0000 7f9f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.148+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197773776 0 0) 0x7f9f3c138020 con 0x7f9f3c0727a0 2024-03-25T19:42:12.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.148+0000 7f9f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.148+0000 7f9f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.148+0000 7f9f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.148+0000 7f9f32ffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:42:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.148+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20004eb0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.149+0000 7f9f32ffd640 10 client.?.objecter ms_handle_connect 0x7f9f3c13fc60 2024-03-25T19:42:12.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.150+0000 7f9f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.160+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636575203 0 0) 0x7f9f3c138340 con 0x7f9f3c070570 2024-03-25T19:42:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.160+0000 7f9f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.160+0000 7f9f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.160+0000 7f9f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.160+0000 7f9f32ffd640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:42:12.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.160+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3c138020 con 0x7f9f3c070570 2024-03-25T19:42:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933745621 0 0) 0x7f9f20004eb0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f3c138340 con 0x7f9f3c0727a0 2024-03-25T19:42:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9f2c001a30 con 0x7f9f3c0727a0 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:12.178 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.177+0000 7f9f32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:12.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.181+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056874613 0 0) 0x7f9f3c0ae1e0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.181+0000 7f9f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.181+0000 7f9f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.181+0000 7f9f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.181+0000 7f9f32ffd640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:42:12.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.181+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20004eb0 con 0x7f9f3c13fc60 2024-03-25T19:42:12.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.193+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179534327 0 0) 0x7f9f3c138020 con 0x7f9f3c070570 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.193+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f3c0ae1e0 con 0x7f9f3c070570 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.193+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9f38001b00 con 0x7f9f3c070570 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.193+0000 7f9f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.193+0000 7f9f32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.193+0000 7f9f32ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:12.194 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:12.195 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:12.195 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:12.195 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3952884937 0 0) 0x7f9f3c138340 con 0x7f9f3c0727a0 2024-03-25T19:42:12.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 >> v1:172.21.15.47:6790/0 conn(0x7f9f3c13fc60 legacy=0x7f9f3c1e2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 >> v1:172.21.15.47:6789/0 conn(0x7f9f3c070570 legacy=0x7f9f3c137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 1 monclient: found mon.b 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0adaf0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 10 monclient: _renew_subs 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c0adda0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 10 monclient: _renew_subs 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c1d8290 con 0x7f9f3c0727a0 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.196+0000 7f9f4472b640 1 librados: init done 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.197+0000 7f9f4472b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.197+0000 7f9f4472b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.197+0000 7f9f4472b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:12.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.197+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c1381b0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f2c001c90 con 0x7f9f3c0727a0 2024-03-25T19:42:12.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:12.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9f2c003a60 con 0x7f9f3c0727a0 2024-03-25T19:42:12.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f32ffd640 10 monclient: dump: 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:12.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.499+0000 7f9f40cc4640 4 set_mon_vals no callback set 2024-03-25T19:42:12.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.500+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f9f2c003c80 con 0x7f9f3c0727a0 2024-03-25T19:42:12.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.501+0000 7f9f424c7640 10 monclient: get_auth_request con 0x7f9f2006dfb0 auth_method 0 2024-03-25T19:42:12.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.501+0000 7f9f32ffd640 10 client.4561.objecter ms_handle_connect 0x7f9f2006dfb0 2024-03-25T19:42:12.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f9f2c054860 con 0x7f9f3c0727a0 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 10 client.4561.objecter ms_dispatch 0x7f9f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f2c050370 con 0x7f9f3c0727a0 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.505+0000 7f9f32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:12.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.720+0000 7f9f4472b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:12.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.720+0000 7f9f4472b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:12.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.720+0000 7f9f4472b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:12.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.720+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f3c1d8540 con 0x7f9f3c0727a0 2024-03-25T19:42:12.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.752+0000 7f9f32ffd640 1 -- 172.21.15.47:0/108885897 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f9f2c0522a0 con 0x7f9f3c0727a0 2024-03-25T19:42:12.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.752+0000 7f9f32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:12.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.752+0000 7f9f32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:12.753 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:12.753 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 10 librados: watch_flush enter 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 10 librados: watch_flush exit 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 10 client.4561.objecter successfully canceled tick 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 >> v1:172.21.15.142:6800/1025302268 conn(0x7f9f2006dfb0 legacy=0x7f9f20070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 10 monclient: shutdown 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 >> v1:172.21.15.142:6789/0 conn(0x7f9f3c0727a0 legacy=0x7f9f3c1d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 shutdown_connections 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 >> 172.21.15.47:0/108885897 conn(0x7f9f3c0b4a80 msgr2=0x7f9f3c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:12.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 shutdown_connections 2024-03-25T19:42:12.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 -- 172.21.15.47:0/108885897 wait complete. 2024-03-25T19:42:12.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:12.755+0000 7f9f4472b640 1 librados: shutdown 2024-03-25T19:42:12.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:19.574 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:19.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.687+0000 7f8b6b99d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.687+0000 7f8b6b99d640 10 monclient: build_initial_monmap 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.687+0000 7f8b6b99d640 10 monclient: monmap: 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:19.687879+0000 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:19.687879+0000 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:19.687 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.687+0000 7f8b6b99d640 1 Processor -- start 2024-03-25T19:42:19.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 1 -- start start 2024-03-25T19:42:19.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient: init 2024-03-25T19:42:19.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b640a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6413f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b641430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64142520 con 0x7f8b640a98f0 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640b28a0 con 0x7f8b641430c0 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640b2a30 con 0x7f8b6413f710 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:19.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.688+0000 7f8b6b99d640 10 monclient(hunting): authenticate will time out at 6054.974753s 2024-03-25T19:42:19.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.694+0000 7f8b69739640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8b640a98f0 0x7f8b640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34824/0 (socket says 172.21.15.47:34824) 2024-03-25T19:42:19.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.694+0000 7f8b69739640 1 -- 172.21.15.47:0/3278480525 learned_addr learned my addr 172.21.15.47:0/3278480525 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:19.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.703+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656007481 0 0) 0x7f8b64142520 con 0x7f8b640a98f0 2024-03-25T19:42:19.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.703+0000 7f8b5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:19.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.703+0000 7f8b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:19.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.703+0000 7f8b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:19.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.703+0000 7f8b5bfff640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:42:19.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.703+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b50003520 con 0x7f8b640a98f0 2024-03-25T19:42:19.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.707+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114663684 0 0) 0x7f8b640b28a0 con 0x7f8b641430c0 2024-03-25T19:42:19.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.707+0000 7f8b5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:19.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.707+0000 7f8b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:19.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.707+0000 7f8b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:19.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.707+0000 7f8b5bfff640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:42:19.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.707+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b64142520 con 0x7f8b641430c0 2024-03-25T19:42:19.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284845323 0 0) 0x7f8b50003520 con 0x7f8b640a98f0 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 >> v1:172.21.15.47:6790/0 conn(0x7f8b6413f710 legacy=0x7f8b641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 >> v1:172.21.15.142:6789/0 conn(0x7f8b641430c0 legacy=0x7f8b6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 1 monclient: found mon.noname-a 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640b44c0 con 0x7f8b640a98f0 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: _finish_auth 0 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8b60001690 con 0x7f8b640a98f0 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:19.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b5bfff640 10 monclient: dump: 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b6b99d640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:42:19.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.711+0000 7f8b6b99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.716+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b600017f0 con 0x7f8b640a98f0 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.716+0000 7f8b5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.716+0000 7f8b5bfff640 1 -- 172.21.15.47:0/3278480525 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8b60001c00 con 0x7f8b640a98f0 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.716+0000 7f8b5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.716+0000 7f8b5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.716+0000 7f8b5bfff640 10 monclient: dump: 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:19.716 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:19.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 4 set_mon_vals no callback set 2024-03-25T19:42:19.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 10 monclient: shutdown 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 1 -- 172.21.15.47:0/3278480525 >> v1:172.21.15.47:6789/0 conn(0x7f8b640a98f0 legacy=0x7f8b640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 1 -- 172.21.15.47:0/3278480525 shutdown_connections 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 1 -- 172.21.15.47:0/3278480525 >> 172.21.15.47:0/3278480525 conn(0x7f8b640b7ea0 msgr2=0x7f8b640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 1 -- 172.21.15.47:0/3278480525 shutdown_connections 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.717+0000 7f8b6b99d640 1 -- 172.21.15.47:0/3278480525 wait complete. 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.718+0000 7f8b6b99d640 10 monclient: build_initial_monmap 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.718+0000 7f8b6b99d640 10 monclient: monmap: 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:19.720 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.718+0000 7f8b6b99d640 1 librados: starting msgr at 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.718+0000 7f8b6b99d640 1 librados: starting objecter 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 Processor -- start 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 -- start start 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 librados: setting wanted keys 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 librados: calling monclient init 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient: init 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:19.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b640a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b6413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b641430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b641369b0 con 0x7f8b6413f710 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b641366b0 con 0x7f8b641430c0 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64133ad0 con 0x7f8b640a98f0 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:19.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.719+0000 7f8b6b99d640 10 monclient(hunting): authenticate will time out at 6055.005743s 2024-03-25T19:42:19.733 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.734+0000 7f8b69f3a640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8b641430c0 0x7f8b641333c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:45110/0 (socket says 172.21.15.47:45110) 2024-03-25T19:42:19.733 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.734+0000 7f8b69f3a640 1 -- 172.21.15.47:0/4191265943 learned_addr learned my addr 172.21.15.47:0/4191265943 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:19.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.735+0000 7f8b59ffb640 10 client.?.objecter ms_handle_connect 0x7f8b641430c0 2024-03-25T19:42:19.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.735+0000 7f8b59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:19.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.737+0000 7f8b59ffb640 10 client.?.objecter ms_handle_connect 0x7f8b640a98f0 2024-03-25T19:42:19.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.737+0000 7f8b59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:19.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.741+0000 7f8b59ffb640 10 client.?.objecter ms_handle_connect 0x7f8b6413f710 2024-03-25T19:42:19.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.741+0000 7f8b59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:19.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.753+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924739527 0 0) 0x7f8b641366b0 con 0x7f8b641430c0 2024-03-25T19:42:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.753+0000 7f8b59ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.753+0000 7f8b59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.753+0000 7f8b59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.753+0000 7f8b59ffb640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:42:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.753+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b48004eb0 con 0x7f8b641430c0 2024-03-25T19:42:19.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.757+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324343927 0 0) 0x7f8b641369b0 con 0x7f8b6413f710 2024-03-25T19:42:19.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.757+0000 7f8b59ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:19.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.757+0000 7f8b59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:19.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.757+0000 7f8b59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:19.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.757+0000 7f8b59ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:42:19.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.757+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b641366b0 con 0x7f8b6413f710 2024-03-25T19:42:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.762+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800061381 0 0) 0x7f8b64133ad0 con 0x7f8b640a98f0 2024-03-25T19:42:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.762+0000 7f8b59ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.762+0000 7f8b59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.762+0000 7f8b59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.762+0000 7f8b59ffb640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:42:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.762+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b641369b0 con 0x7f8b640a98f0 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.770+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280689008 0 0) 0x7f8b48004eb0 con 0x7f8b641430c0 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.770+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b64133ad0 con 0x7f8b641430c0 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.770+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8b5c001c70 con 0x7f8b641430c0 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.770+0000 7f8b59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.770+0000 7f8b59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.771+0000 7f8b59ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:19.770 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.771+0000 7f8b59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:19.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.785+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3944780972 0 0) 0x7f8b641366b0 con 0x7f8b6413f710 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.785+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b48004eb0 con 0x7f8b6413f710 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.785+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8b540018b0 con 0x7f8b6413f710 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.785+0000 7f8b59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.785+0000 7f8b59ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.785+0000 7f8b59ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:19.785 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:19.786 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:19.786 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.801+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678477561 0 0) 0x7f8b641369b0 con 0x7f8b640a98f0 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.801+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b48005ed0 con 0x7f8b640a98f0 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.801+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8b60001b00 con 0x7f8b640a98f0 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.801+0000 7f8b59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.801+0000 7f8b59ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.801+0000 7f8b59ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:19.801 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:19.802 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:19.802 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:19.802 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:19.802 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:19.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1542944166 0 0) 0x7f8b64133ad0 con 0x7f8b641430c0 2024-03-25T19:42:19.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 >> v1:172.21.15.47:6790/0 conn(0x7f8b640a98f0 legacy=0x7f8b641371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 >> v1:172.21.15.47:6789/0 conn(0x7f8b6413f710 legacy=0x7f8b64132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:19.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 1 monclient: found mon.b 2024-03-25T19:42:19.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64136e40 con 0x7f8b641430c0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 10 monclient: _renew_subs 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b641378f0 con 0x7f8b641430c0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 10 monclient: _renew_subs 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b640b1480 con 0x7f8b641430c0 2024-03-25T19:42:19.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.803+0000 7f8b6b99d640 1 librados: init done 2024-03-25T19:42:19.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.805+0000 7f8b6b99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:19.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.805+0000 7f8b6b99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:19.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.805+0000 7f8b6b99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:19.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:19.805+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b64137030 con 0x7f8b641430c0 2024-03-25T19:42:20.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b5c003990 con 0x7f8b641430c0 2024-03-25T19:42:20.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:20.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8b5c003d60 con 0x7f8b641430c0 2024-03-25T19:42:20.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:20.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b59ffb640 10 monclient: dump: 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:20.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.079+0000 7f8b5b7fe640 4 set_mon_vals no callback set 2024-03-25T19:42:20.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.080+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f8b5c003f80 con 0x7f8b641430c0 2024-03-25T19:42:20.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.081+0000 7f8b69739640 10 monclient: get_auth_request con 0x7f8b4806e480 auth_method 0 2024-03-25T19:42:20.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.081+0000 7f8b59ffb640 10 client.4567.objecter ms_handle_connect 0x7f8b4806e480 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.085+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f8b5c054ad0 con 0x7f8b641430c0 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.085+0000 7f8b59ffb640 10 client.4567.objecter ms_dispatch 0x7f8b64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.085+0000 7f8b59ffb640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.085+0000 7f8b59ffb640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.085+0000 7f8b59ffb640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.086+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b5c0505e0 con 0x7f8b641430c0 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.086+0000 7f8b59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.086+0000 7f8b59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:20.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.348+0000 7f8b6b99d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:20.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.348+0000 7f8b6b99d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:20.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.348+0000 7f8b6b99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:20.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.348+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b64136b60 con 0x7f8b641430c0 2024-03-25T19:42:20.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.371+0000 7f8b59ffb640 1 -- 172.21.15.47:0/4191265943 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f8b64136b60 con 0x7f8b641430c0 2024-03-25T19:42:20.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.371+0000 7f8b59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:20.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.371+0000 7f8b59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:20.371 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:20.371 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:20.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.373+0000 7f8b6b99d640 10 librados: watch_flush enter 2024-03-25T19:42:20.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.373+0000 7f8b6b99d640 10 librados: watch_flush exit 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.373+0000 7f8b6b99d640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.373+0000 7f8b6b99d640 10 client.4567.objecter successfully canceled tick 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 >> v1:172.21.15.142:6800/1025302268 conn(0x7f8b4806e480 legacy=0x7f8b48070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 10 monclient: shutdown 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 >> v1:172.21.15.142:6789/0 conn(0x7f8b641430c0 legacy=0x7f8b641333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 shutdown_connections 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 >> 172.21.15.47:0/4191265943 conn(0x7f8b640b7ea0 msgr2=0x7f8b64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 shutdown_connections 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 -- 172.21.15.47:0/4191265943 wait complete. 2024-03-25T19:42:20.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:20.374+0000 7f8b6b99d640 1 librados: shutdown 2024-03-25T19:42:20.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:23.472 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:00.708 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.478 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:00.714 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.522 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:00.712 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.523 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:00.759 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.528 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:00.764 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.532 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:00.722 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.541 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:00.777 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.563 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:00.753 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:23.565 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:00.801 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.192 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.310+0000 7ffae55f4640 10 monclient: get_monmap_and_config 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.310+0000 7ffae55f4640 10 monclient: build_initial_monmap 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.310+0000 7ffae55f4640 10 monclient: monmap: 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:27.311189+0000 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:27.311189+0000 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:27.310 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:27.311 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 1 Processor -- start 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 1 -- start start 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient: init 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient(hunting): picked mon.noname-c con 0x7ffae00b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient(hunting): picked mon.noname-b con 0x7ffae00b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient(hunting): picked mon.noname-a con 0x7ffae014be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00af940 con 0x7ffae014be90 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00b0a70 con 0x7ffae00b4d60 2024-03-25T19:42:27.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00b0d50 con 0x7ffae00b1730 2024-03-25T19:42:27.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:27.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.311+0000 7ffae55f4640 10 monclient(hunting): authenticate will time out at 6062.597630s 2024-03-25T19:42:27.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.314+0000 7ffad67fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ffae00b1730 0x7ffae00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:36116/0 (socket says 172.21.15.47:36116) 2024-03-25T19:42:27.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.314+0000 7ffad67fc640 1 -- 172.21.15.47:0/1736265064 learned_addr learned my addr 172.21.15.47:0/1736265064 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:27.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.382+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480828097 0 0) 0x7ffae00b0d50 con 0x7ffae00b1730 2024-03-25T19:42:27.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.382+0000 7ffade7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.382+0000 7ffade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.382+0000 7ffade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.382+0000 7ffade7fc640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:42:27.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.382+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffacc003520 con 0x7ffae00b1730 2024-03-25T19:42:27.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.385+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793103863 0 0) 0x7ffae00b0a70 con 0x7ffae00b4d60 2024-03-25T19:42:27.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.385+0000 7ffade7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.385+0000 7ffade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.385+0000 7ffade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.385+0000 7ffade7fc640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:42:27.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.385+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae00b0d50 con 0x7ffae00b4d60 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10650967 0 0) 0x7ffacc003520 con 0x7ffae00b1730 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 >> v1:172.21.15.142:6789/0 conn(0x7ffae00b4d60 legacy=0x7ffae013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 >> v1:172.21.15.47:6789/0 conn(0x7ffae014be90 legacy=0x7ffae014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 1 monclient: found mon.noname-b 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae00b1030 con 0x7ffae00b1730 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:27.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ffac8001690 con 0x7ffae00b1730 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffade7fc640 10 monclient: dump: 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:27.392 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:27.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffae55f4640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:42:27.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.391+0000 7ffae55f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:27.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.405+0000 7ffade7fc640 1 -- 172.21.15.47:0/1736265064 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffac80017f0 con 0x7ffae00b1730 2024-03-25T19:42:27.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.405+0000 7ffade7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:27.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.405+0000 7ffae55f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:27.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.405+0000 7ffae55f4640 4 set_mon_vals no callback set 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 10 monclient: shutdown 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 1 -- 172.21.15.47:0/1736265064 >> v1:172.21.15.47:6790/0 conn(0x7ffae00b1730 legacy=0x7ffae00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 1 -- 172.21.15.47:0/1736265064 shutdown_connections 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 1 -- 172.21.15.47:0/1736265064 >> 172.21.15.47:0/1736265064 conn(0x7ffae00617a0 msgr2=0x7ffae0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 1 -- 172.21.15.47:0/1736265064 shutdown_connections 2024-03-25T19:42:27.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.406+0000 7ffae55f4640 1 -- 172.21.15.47:0/1736265064 wait complete. 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 10 monclient: build_initial_monmap 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 10 monclient: monmap: 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:27.407 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 1 librados: starting msgr at 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 1 librados: starting objecter 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 1 Processor -- start 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 1 -- start start 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 1 librados: setting wanted keys 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 1 librados: calling monclient init 2024-03-25T19:42:27.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.407+0000 7ffae55f4640 10 monclient: init 2024-03-25T19:42:27.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:27.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:27.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient(hunting): picked mon.noname-a con 0x7ffae00b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:27.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient(hunting): picked mon.noname-c con 0x7ffae00b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:27.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient(hunting): picked mon.noname-b con 0x7ffae014be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:27.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae01dabc0 con 0x7ffae00b1730 2024-03-25T19:42:27.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae0140850 con 0x7ffae014be90 2024-03-25T19:42:27.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae013d0f0 con 0x7ffae00b4d60 2024-03-25T19:42:27.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:27.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.408+0000 7ffae55f4640 10 monclient(hunting): authenticate will time out at 6062.694752s 2024-03-25T19:42:27.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.414+0000 7ffadf7fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ffae014be90 0x7ffae0141310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:49824/0 (socket says 172.21.15.47:49824) 2024-03-25T19:42:27.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.414+0000 7ffadf7fe640 1 -- 172.21.15.47:0/1788604827 learned_addr learned my addr 172.21.15.47:0/1788604827 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:27.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.416+0000 7ffad7fff640 10 client.?.objecter ms_handle_connect 0x7ffae014be90 2024-03-25T19:42:27.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.416+0000 7ffad7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:27.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.422+0000 7ffad7fff640 10 client.?.objecter ms_handle_connect 0x7ffae00b1730 2024-03-25T19:42:27.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.422+0000 7ffad7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:27.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.427+0000 7ffad7fff640 10 client.?.objecter ms_handle_connect 0x7ffae00b4d60 2024-03-25T19:42:27.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.427+0000 7ffad7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:27.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.431+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912599327 0 0) 0x7ffae0140850 con 0x7ffae014be90 2024-03-25T19:42:27.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.431+0000 7ffad7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.431+0000 7ffad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.431+0000 7ffad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.431+0000 7ffad7fff640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:42:27.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.432+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac0004eb0 con 0x7ffae014be90 2024-03-25T19:42:27.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.433+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717801762 0 0) 0x7ffae01dabc0 con 0x7ffae00b1730 2024-03-25T19:42:27.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.433+0000 7ffad7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.433+0000 7ffad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.433+0000 7ffad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.433+0000 7ffad7fff640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:42:27.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.433+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae0140850 con 0x7ffae00b1730 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437865127 0 0) 0x7ffae013d0f0 con 0x7ffae00b4d60 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae01dabc0 con 0x7ffae00b4d60 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3580044237 0 0) 0x7ffae0140850 con 0x7ffae00b1730 2024-03-25T19:42:27.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae013d0f0 con 0x7ffae00b1730 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ffad00018b0 con 0x7ffae00b1730 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:27.447 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:27.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.446+0000 7ffad7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.449+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3324039285 0 0) 0x7ffac0004eb0 con 0x7ffae014be90 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.449+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae0140850 con 0x7ffae014be90 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.450+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ffad8001ad0 con 0x7ffae014be90 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.450+0000 7ffad7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.450+0000 7ffad7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.450+0000 7ffad7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.449 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.450 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.450 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:27.450 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:27.450 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:27.450 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:27.450 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:27.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3804706560 0 0) 0x7ffae0140850 con 0x7ffae014be90 2024-03-25T19:42:27.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 >> v1:172.21.15.47:6790/0 conn(0x7ffae00b4d60 legacy=0x7ffae0132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 >> v1:172.21.15.47:6789/0 conn(0x7ffae00b1730 legacy=0x7ffae01db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:27.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:27.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 1 monclient: found mon.b 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae013d280 con 0x7ffae014be90 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffad7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 10 monclient: _renew_subs 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae01db7f0 con 0x7ffae014be90 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 10 monclient: _renew_subs 2024-03-25T19:42:27.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:27.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae0133680 con 0x7ffae014be90 2024-03-25T19:42:27.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.457+0000 7ffae55f4640 1 librados: init done 2024-03-25T19:42:27.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.458+0000 7ffae55f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:27.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.458+0000 7ffae55f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:27.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.458+0000 7ffae55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:27.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.458+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae00627b0 con 0x7ffae014be90 2024-03-25T19:42:27.733 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad8001fe0 con 0x7ffae014be90 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffad7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ffad8004d80 con 0x7ffae014be90 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffad7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffad7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffad7fff640 10 monclient: dump: 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:27.734 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.734+0000 7ffaddffb640 4 set_mon_vals no callback set 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.735+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ffad801ad50 con 0x7ffae014be90 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.736+0000 7ffadeffd640 10 monclient: get_auth_request con 0x7ffac006df70 auth_method 0 2024-03-25T19:42:27.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.736+0000 7ffad7fff640 10 client.4573.objecter ms_handle_connect 0x7ffac006df70 2024-03-25T19:42:27.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7ffad80549a0 con 0x7ffae014be90 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 10 client.4573.objecter ms_dispatch 0x7ffae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffad80502e0 con 0x7ffae014be90 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:27.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.740+0000 7ffad7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.992+0000 7ffae55f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.992+0000 7ffae55f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.992+0000 7ffae55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:27.992+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffae0069f40 con 0x7ffae014be90 2024-03-25T19:42:28.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.029+0000 7ffad7fff640 1 -- 172.21.15.47:0/1788604827 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7ffad80523e0 con 0x7ffae014be90 2024-03-25T19:42:28.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.029+0000 7ffad7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.029+0000 7ffad7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:28.029 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:28.029 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:28.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 10 librados: watch_flush enter 2024-03-25T19:42:28.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 10 librados: watch_flush exit 2024-03-25T19:42:28.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T19:42:28.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 10 client.4573.objecter successfully canceled tick 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 >> v1:172.21.15.142:6800/1025302268 conn(0x7ffac006df70 legacy=0x7ffac0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 10 monclient: shutdown 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 >> v1:172.21.15.142:6789/0 conn(0x7ffae014be90 legacy=0x7ffae0141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 shutdown_connections 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 >> 172.21.15.47:0/1788604827 conn(0x7ffae00617a0 msgr2=0x7ffae00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 shutdown_connections 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 -- 172.21.15.47:0/1788604827 wait complete. 2024-03-25T19:42:28.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:28.032+0000 7ffae55f4640 1 librados: shutdown 2024-03-25T19:42:28.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:34.851 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:34.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.965+0000 7fa4edb5d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:34.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.965+0000 7fa4edb5d640 10 monclient: build_initial_monmap 2024-03-25T19:42:34.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.965+0000 7fa4edb5d640 10 monclient: monmap: 2024-03-25T19:42:34.965 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:34.965 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:34.965 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:34.966544+0000 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:34.966544+0000 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:34.966 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:34.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.966+0000 7fa4edb5d640 1 Processor -- start 2024-03-25T19:42:34.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.966+0000 7fa4edb5d640 1 -- start start 2024-03-25T19:42:34.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.966+0000 7fa4edb5d640 10 monclient: init 2024-03-25T19:42:34.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e80b10d0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e80b4360 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e80a9470 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e80a9ab0 con 0x7fa4e80a9470 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e80b2400 con 0x7fa4e80b4360 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e80aa2a0 con 0x7fa4e80b10d0 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:34.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.967+0000 7fa4edb5d640 10 monclient(hunting): authenticate will time out at 6070.253338s 2024-03-25T19:42:34.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.972+0000 7fa4e77fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa4e80b10d0 0x7fa4e80b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:37854/0 (socket says 172.21.15.47:37854) 2024-03-25T19:42:34.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.972+0000 7fa4e77fe640 1 -- 172.21.15.47:0/3637797733 learned_addr learned my addr 172.21.15.47:0/3637797733 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:34.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.986+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572150044 0 0) 0x7fa4e80aa2a0 con 0x7fa4e80b10d0 2024-03-25T19:42:34.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.986+0000 7fa4e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:34.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.986+0000 7fa4e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:34.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.986+0000 7fa4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:34.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.986+0000 7fa4e67fc640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:42:34.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.986+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d0003520 con 0x7fa4e80b10d0 2024-03-25T19:42:34.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.997+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122545903 0 0) 0x7fa4e80a9ab0 con 0x7fa4e80a9470 2024-03-25T19:42:34.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.997+0000 7fa4e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:34.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.997+0000 7fa4e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:34.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.997+0000 7fa4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:34.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.997+0000 7fa4e67fc640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:42:34.997 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:34.997+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e80aa2a0 con 0x7fa4e80a9470 2024-03-25T19:42:35.004 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.004+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830789745 0 0) 0x7fa4e80b2400 con 0x7fa4e80b4360 2024-03-25T19:42:35.004 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.004+0000 7fa4e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.004 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.004+0000 7fa4e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.004 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.004+0000 7fa4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.004 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.004+0000 7fa4e67fc640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:42:35.004 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.004+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e80a9ab0 con 0x7fa4e80b4360 2024-03-25T19:42:35.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118443506 0 0) 0x7fa4d0003520 con 0x7fa4e80b10d0 2024-03-25T19:42:35.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 >> v1:172.21.15.142:6789/0 conn(0x7fa4e80b4360 legacy=0x7fa4e80b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 >> v1:172.21.15.47:6789/0 conn(0x7fa4e80a9470 legacy=0x7fa4e814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 1 monclient: found mon.noname-b 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e80b14b0 con 0x7fa4e80b10d0 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa4d8001690 con 0x7fa4e80b10d0 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4edb5d640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4edb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:35.015 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4e67fc640 10 monclient: dump: 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.016 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.014+0000 7fa4edb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:35.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4d80017f0 con 0x7fa4e80b10d0 2024-03-25T19:42:35.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4e67fc640 1 -- 172.21.15.47:0/3637797733 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa4d8001c40 con 0x7fa4e80b10d0 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4edb5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4edb5d640 4 set_mon_vals no callback set 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.045+0000 7fa4e67fc640 10 monclient: dump: 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:35.045 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:35.046 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:35.046 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:35.046 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 10 monclient: shutdown 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3637797733 >> v1:172.21.15.47:6790/0 conn(0x7fa4e80b10d0 legacy=0x7fa4e80b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3637797733 shutdown_connections 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3637797733 >> 172.21.15.47:0/3637797733 conn(0x7fa4e80617a0 msgr2=0x7fa4e8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3637797733 shutdown_connections 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.046+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3637797733 wait complete. 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.047+0000 7fa4edb5d640 10 monclient: build_initial_monmap 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.047+0000 7fa4edb5d640 10 monclient: monmap: 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:35.048 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.047+0000 7fa4edb5d640 1 librados: starting msgr at 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.047+0000 7fa4edb5d640 1 librados: starting objecter 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.047+0000 7fa4edb5d640 1 Processor -- start 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 1 -- start start 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 1 librados: setting wanted keys 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 1 librados: calling monclient init 2024-03-25T19:42:35.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient: init 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e80a9470 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e80b10d0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e80b4360 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e81427d0 con 0x7fa4e80b4360 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e813b090 con 0x7fa4e80a9470 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e81db570 con 0x7fa4e80b10d0 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:35.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.048+0000 7fa4edb5d640 10 monclient(hunting): authenticate will time out at 6070.334528s 2024-03-25T19:42:35.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.057+0000 7fa4e7fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa4e80b4360 0x7fa4e8133050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54966/0 (socket says 172.21.15.47:54966) 2024-03-25T19:42:35.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.057+0000 7fa4e7fff640 1 -- 172.21.15.47:0/3463066096 learned_addr learned my addr 172.21.15.47:0/3463066096 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:35.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.058+0000 7fa4cffff640 10 client.?.objecter ms_handle_connect 0x7fa4e80b4360 2024-03-25T19:42:35.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.058+0000 7fa4cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.061+0000 7fa4cffff640 10 client.?.objecter ms_handle_connect 0x7fa4e80b10d0 2024-03-25T19:42:35.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.061+0000 7fa4cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.065+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843213357 0 0) 0x7fa4e81427d0 con 0x7fa4e80b4360 2024-03-25T19:42:35.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.065+0000 7fa4cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.065+0000 7fa4cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.065+0000 7fa4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.065+0000 7fa4cffff640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:42:35.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.065+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d4004eb0 con 0x7fa4e80b4360 2024-03-25T19:42:35.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.069+0000 7fa4cffff640 10 client.?.objecter ms_handle_connect 0x7fa4e80a9470 2024-03-25T19:42:35.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.069+0000 7fa4cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.070+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596387260 0 0) 0x7fa4e81db570 con 0x7fa4e80b10d0 2024-03-25T19:42:35.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.070+0000 7fa4cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.070+0000 7fa4cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.070+0000 7fa4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.070+0000 7fa4cffff640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:42:35.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.070+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e81427d0 con 0x7fa4e80b10d0 2024-03-25T19:42:35.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.074+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261162536 0 0) 0x7fa4e813b090 con 0x7fa4e80a9470 2024-03-25T19:42:35.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.074+0000 7fa4cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.074+0000 7fa4cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.074+0000 7fa4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.074+0000 7fa4cffff640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:42:35.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.074+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e81db570 con 0x7fa4e80a9470 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342915239 0 0) 0x7fa4e81427d0 con 0x7fa4e80b10d0 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4e813b090 con 0x7fa4e80b10d0 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa4dc0017d0 con 0x7fa4e80b10d0 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 10 monclient(hunting): dump: 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:35.080 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.081 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.080+0000 7fa4cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.081+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295523944 0 0) 0x7fa4d4004eb0 con 0x7fa4e80b4360 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.081+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4e81427d0 con 0x7fa4e80b4360 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.081+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa4e0001dd0 con 0x7fa4e80b4360 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.081+0000 7fa4cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.081+0000 7fa4cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.081+0000 7fa4cffff640 10 monclient(hunting): dump: 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:35.082 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.083+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4106479015 0 0) 0x7fa4e81db570 con 0x7fa4e80a9470 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.083+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d4004eb0 con 0x7fa4e80a9470 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.083+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa4d8001b00 con 0x7fa4e80a9470 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.083+0000 7fa4cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.083+0000 7fa4cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.083+0000 7fa4cffff640 10 monclient(hunting): dump: 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:35.084 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1673210825 0 0) 0x7fa4d4004eb0 con 0x7fa4e80a9470 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 >> v1:172.21.15.47:6790/0 conn(0x7fa4e80b10d0 legacy=0x7fa4e81dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 >> v1:172.21.15.47:6789/0 conn(0x7fa4e80b4360 legacy=0x7fa4e8133050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 1 monclient: found mon.b 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e81da940 con 0x7fa4e80a9470 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:42:35.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 10 monclient: _renew_subs 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e8134840 con 0x7fa4e80a9470 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 10 monclient: _renew_subs 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e8141820 con 0x7fa4e80a9470 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.091+0000 7fa4edb5d640 1 librados: init done 2024-03-25T19:42:35.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.092+0000 7fa4edb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.092+0000 7fa4edb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.092+0000 7fa4edb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:35.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.092+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e81da730 con 0x7fa4e80a9470 2024-03-25T19:42:35.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4d80011c0 con 0x7fa4e80a9470 2024-03-25T19:42:35.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa4d8004bf0 con 0x7fa4e80a9470 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4cffff640 10 monclient: dump: 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:35.366 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:35.367 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.366+0000 7fa4e5ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.367+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fa4d801abc0 con 0x7fa4e80a9470 2024-03-25T19:42:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.367+0000 7fa4e6ffd640 10 monclient: get_auth_request con 0x7fa4d406e130 auth_method 0 2024-03-25T19:42:35.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.367+0000 7fa4cffff640 10 client.4579.objecter ms_handle_connect 0x7fa4d406e130 2024-03-25T19:42:35.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.372+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fa4d8054790 con 0x7fa4e80a9470 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.372+0000 7fa4cffff640 10 client.4579.objecter ms_dispatch 0x7fa4e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.372+0000 7fa4cffff640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.372+0000 7fa4cffff640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.372+0000 7fa4cffff640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.373+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4d80502a0 con 0x7fa4e80a9470 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.373+0000 7fa4cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.373+0000 7fa4cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:35.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.613+0000 7fa4edb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:35.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.613+0000 7fa4edb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:35.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.613+0000 7fa4edb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:35.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.613+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4e81daad0 con 0x7fa4e80a9470 2024-03-25T19:42:35.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.630+0000 7fa4cffff640 1 -- 172.21.15.47:0/3463066096 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fa4d80521d0 con 0x7fa4e80a9470 2024-03-25T19:42:35.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.630+0000 7fa4cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:35.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.630+0000 7fa4cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:35.630 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:35.630 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 10 librados: watch_flush enter 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 10 librados: watch_flush exit 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 10 client.4579.objecter successfully canceled tick 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 >> v1:172.21.15.142:6800/1025302268 conn(0x7fa4d406e130 legacy=0x7fa4d40705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 10 monclient: shutdown 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:35.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 >> v1:172.21.15.142:6789/0 conn(0x7fa4e80a9470 legacy=0x7fa4e81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 shutdown_connections 2024-03-25T19:42:35.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 >> 172.21.15.47:0/3463066096 conn(0x7fa4e80617a0 msgr2=0x7fa4e80b61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:35.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 shutdown_connections 2024-03-25T19:42:35.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 -- 172.21.15.47:0/3463066096 wait complete. 2024-03-25T19:42:35.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:35.633+0000 7fa4edb5d640 1 librados: shutdown 2024-03-25T19:42:35.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:42.452 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:42.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.567+0000 7f9a3e263640 10 monclient: get_monmap_and_config 2024-03-25T19:42:42.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.567+0000 7f9a3e263640 10 monclient: build_initial_monmap 2024-03-25T19:42:42.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.567+0000 7f9a3e263640 10 monclient: monmap: 2024-03-25T19:42:42.567 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:42.567 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:42.567 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:42.568531+0000 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:42.568531+0000 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:42.568 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.568+0000 7f9a3e263640 1 Processor -- start 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 1 -- start start 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 10 monclient: init 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 10 monclient(hunting): picked mon.noname-c con 0x7f9a380a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 10 monclient(hunting): picked mon.noname-a con 0x7f9a3813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 10 monclient(hunting): picked mon.noname-b con 0x7f9a381430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a38142520 con 0x7f9a3813f710 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380b28a0 con 0x7f9a380a98f0 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.569+0000 7f9a3e263640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380b2a30 con 0x7f9a381430c0 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.570+0000 7f9a3e263640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:42.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.570+0000 7f9a3e263640 10 monclient(hunting): authenticate will time out at 6077.855963s 2024-03-25T19:42:42.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.581+0000 7f9a37fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9a380a98f0 0x7f9a380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:51188/0 (socket says 172.21.15.47:51188) 2024-03-25T19:42:42.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.581+0000 7f9a37fff640 1 -- 172.21.15.47:0/964517400 learned_addr learned my addr 172.21.15.47:0/964517400 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:42.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.588+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299911389 0 0) 0x7f9a380b2a30 con 0x7f9a381430c0 2024-03-25T19:42:42.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.588+0000 7f9a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.588+0000 7f9a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.588+0000 7f9a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.588+0000 7f9a36ffd640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:42:42.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.588+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a20003520 con 0x7f9a381430c0 2024-03-25T19:42:42.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.591+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955148416 0 0) 0x7f9a380b28a0 con 0x7f9a380a98f0 2024-03-25T19:42:42.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.591+0000 7f9a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.591+0000 7f9a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.591+0000 7f9a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.591+0000 7f9a36ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:42:42.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.591+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a380b2a30 con 0x7f9a380a98f0 2024-03-25T19:42:42.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.604+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087301625 0 0) 0x7f9a38142520 con 0x7f9a3813f710 2024-03-25T19:42:42.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.604+0000 7f9a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.604+0000 7f9a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.604+0000 7f9a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.604+0000 7f9a36ffd640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:42:42.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.604+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a380b28a0 con 0x7f9a3813f710 2024-03-25T19:42:42.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527327763 0 0) 0x7f9a20003520 con 0x7f9a381430c0 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 >> v1:172.21.15.142:6789/0 conn(0x7f9a380a98f0 legacy=0x7f9a380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 >> v1:172.21.15.47:6789/0 conn(0x7f9a3813f710 legacy=0x7f9a381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 1 monclient: found mon.noname-b 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380b44c0 con 0x7f9a381430c0 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9a30001a10 con 0x7f9a381430c0 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a3e263640 5 monclient: authenticate success, global_id 4571 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a3e263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:42.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a36ffd640 10 monclient: dump: 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.607+0000 7f9a3e263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:42.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a30001b70 con 0x7f9a381430c0 2024-03-25T19:42:42.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:42.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a36ffd640 1 -- 172.21.15.47:0/964517400 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9a30001f80 con 0x7f9a381430c0 2024-03-25T19:42:42.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a36ffd640 10 monclient: dump: 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:42.642 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:42.643 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:42.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 4 set_mon_vals no callback set 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 10 monclient: shutdown 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 1 -- 172.21.15.47:0/964517400 >> v1:172.21.15.47:6790/0 conn(0x7f9a381430c0 legacy=0x7f9a3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 1 -- 172.21.15.47:0/964517400 shutdown_connections 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.642+0000 7f9a3e263640 1 -- 172.21.15.47:0/964517400 >> 172.21.15.47:0/964517400 conn(0x7f9a380b7ea0 msgr2=0x7f9a380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.643+0000 7f9a3e263640 1 -- 172.21.15.47:0/964517400 shutdown_connections 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.643+0000 7f9a3e263640 1 -- 172.21.15.47:0/964517400 wait complete. 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.643+0000 7f9a3e263640 10 monclient: build_initial_monmap 2024-03-25T19:42:42.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.643+0000 7f9a3e263640 10 monclient: monmap: 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 1 librados: starting msgr at 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 1 librados: starting objecter 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 1 Processor -- start 2024-03-25T19:42:42.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 1 -- start start 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 1 librados: setting wanted keys 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 1 librados: calling monclient init 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.644+0000 7f9a3e263640 10 monclient: init 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient(hunting): picked mon.noname-a con 0x7f9a380a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient(hunting): picked mon.noname-c con 0x7f9a381430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3813c9b0 con 0x7f9a380a98f0 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a381397c0 con 0x7f9a3813f710 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a38139950 con 0x7f9a381430c0 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:42.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.645+0000 7f9a3e263640 10 monclient(hunting): authenticate will time out at 6077.931286s 2024-03-25T19:42:42.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.651+0000 7f9a377fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9a3813f710 0x7f9a381dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:51238/0 (socket says 172.21.15.47:51238) 2024-03-25T19:42:42.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.651+0000 7f9a377fe640 1 -- 172.21.15.47:0/1122918721 learned_addr learned my addr 172.21.15.47:0/1122918721 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:42.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.651+0000 7f9a34ff9640 10 client.?.objecter ms_handle_connect 0x7f9a3813f710 2024-03-25T19:42:42.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.651+0000 7f9a34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:42.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.658+0000 7f9a34ff9640 10 client.?.objecter ms_handle_connect 0x7f9a381430c0 2024-03-25T19:42:42.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.658+0000 7f9a34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:42.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.661+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400483231 0 0) 0x7f9a38139950 con 0x7f9a381430c0 2024-03-25T19:42:42.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.661+0000 7f9a34ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.661+0000 7f9a34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.661+0000 7f9a34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.661+0000 7f9a34ff9640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:42:42.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.661+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a24004eb0 con 0x7f9a381430c0 2024-03-25T19:42:42.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.666+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146945385 0 0) 0x7f9a381397c0 con 0x7f9a3813f710 2024-03-25T19:42:42.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.666+0000 7f9a34ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:42.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.666+0000 7f9a34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:42.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.666+0000 7f9a34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:42.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.666+0000 7f9a34ff9640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:42:42.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.667+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a38139950 con 0x7f9a3813f710 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086230413 0 0) 0x7f9a38139950 con 0x7f9a3813f710 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a381397c0 con 0x7f9a3813f710 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9a2c0019f0 con 0x7f9a3813f710 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.679 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.679+0000 7f9a34ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:42.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.682+0000 7f9a34ff9640 10 client.?.objecter ms_handle_connect 0x7f9a380a98f0 2024-03-25T19:42:42.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.682+0000 7f9a34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868021297 0 0) 0x7f9a24004eb0 con 0x7f9a381430c0 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a38139950 con 0x7f9a381430c0 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9a30001e80 con 0x7f9a381430c0 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:42.685 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:42.686 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2723571975 0 0) 0x7f9a381397c0 con 0x7f9a3813f710 2024-03-25T19:42:42.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 >> v1:172.21.15.47:6790/0 conn(0x7f9a381430c0 legacy=0x7f9a381e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 >> v1:172.21.15.47:6789/0 conn(0x7f9a380a98f0 legacy=0x7f9a38130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 monclient: found mon.b 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38131680 con 0x7f9a3813f710 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 10 monclient: _renew_subs 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a38131950 con 0x7f9a3813f710 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 10 monclient: _renew_subs 2024-03-25T19:42:42.688 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a38135a30 con 0x7f9a3813f710 2024-03-25T19:42:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.686+0000 7f9a3e263640 1 librados: init done 2024-03-25T19:42:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.687+0000 7f9a3e263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.687+0000 7f9a3e263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.687+0000 7f9a3e263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.687+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3813c810 con 0x7f9a3813f710 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.945+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a2c003610 con 0x7f9a3813f710 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.945+0000 7f9a34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:42.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.945+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f9a2c003840 con 0x7f9a3813f710 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.945+0000 7f9a34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.946+0000 7f9a34ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.946+0000 7f9a34ff9640 10 monclient: dump: 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:42.946 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.946+0000 7f9a367fc640 4 set_mon_vals no callback set 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.946+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f9a2c003a60 con 0x7f9a3813f710 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.947+0000 7f9a37fff640 10 monclient: get_auth_request con 0x7f9a2406de40 auth_method 0 2024-03-25T19:42:42.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.948+0000 7f9a34ff9640 10 client.4585.objecter ms_handle_connect 0x7f9a2406de40 2024-03-25T19:42:42.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.953+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f9a2c004460 con 0x7f9a3813f710 2024-03-25T19:42:42.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 10 client.4585.objecter ms_dispatch 0x7f9a38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:42.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:42.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:42.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T19:42:42.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a2c050110 con 0x7f9a3813f710 2024-03-25T19:42:42.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:42.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:42.954+0000 7f9a34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:43.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.214+0000 7f9a3e263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:43.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.214+0000 7f9a3e263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:43.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.214+0000 7f9a3e263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:42:43.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.214+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a38136fc0 con 0x7f9a3813f710 2024-03-25T19:42:43.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.238+0000 7f9a34ff9640 1 -- 172.21.15.47:0/1122918721 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f9a2c052040 con 0x7f9a3813f710 2024-03-25T19:42:43.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.239+0000 7f9a34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:43.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.239+0000 7f9a34ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:43.238 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:43.239 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 10 librados: watch_flush enter 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 10 librados: watch_flush exit 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 10 client.4585.objecter successfully canceled tick 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 >> v1:172.21.15.142:6800/1025302268 conn(0x7f9a2406de40 legacy=0x7f9a24070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 10 monclient: shutdown 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 >> v1:172.21.15.142:6789/0 conn(0x7f9a3813f710 legacy=0x7f9a381dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 shutdown_connections 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 >> 172.21.15.47:0/1122918721 conn(0x7f9a380b7ea0 msgr2=0x7f9a3814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 shutdown_connections 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 -- 172.21.15.47:0/1122918721 wait complete. 2024-03-25T19:42:43.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:43.241+0000 7f9a3e263640 1 librados: shutdown 2024-03-25T19:42:43.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:43.784 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:21.020 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.796 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:21.032 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.831 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:21.066 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.844 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:21.034 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.847 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:21.083 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.849 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:21.039 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.857 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:21.093 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.871 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:21.061 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.890 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:21.126 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:50.061 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.177+0000 7fad09725640 10 monclient: get_monmap_and_config 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.177+0000 7fad09725640 10 monclient: build_initial_monmap 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.177+0000 7fad09725640 10 monclient: monmap: 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:50.179449+0000 2024-03-25T19:42:50.178 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:50.179449+0000 2024-03-25T19:42:50.179 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:50.179 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:50.179 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:50.179 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:50.179 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:50.179 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:50.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.178+0000 7fad09725640 1 Processor -- start 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.178+0000 7fad09725640 1 -- start start 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.178+0000 7fad09725640 10 monclient: init 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient(hunting): picked mon.noname-c con 0x7fad040a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient(hunting): picked mon.noname-a con 0x7fad0413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient(hunting): picked mon.noname-b con 0x7fad041430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad04142520 con 0x7fad0413f710 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad040b28a0 con 0x7fad040a98f0 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad040b2a30 con 0x7fad041430c0 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:50.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.179+0000 7fad09725640 10 monclient(hunting): authenticate will time out at 6085.466883s 2024-03-25T19:42:50.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.184+0000 7fad02ffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fad040a98f0 0x7fad040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:43890/0 (socket says 172.21.15.47:43890) 2024-03-25T19:42:50.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.184+0000 7fad02ffd640 1 -- 172.21.15.47:0/373800178 learned_addr learned my addr 172.21.15.47:0/373800178 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:50.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.202+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741770098 0 0) 0x7fad040b28a0 con 0x7fad040a98f0 2024-03-25T19:42:50.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.202+0000 7fad027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.202+0000 7fad027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.202+0000 7fad027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.202+0000 7fad027fc640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:42:50.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.202+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face4003520 con 0x7fad040a98f0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3634560544 0 0) 0x7face4003520 con 0x7fad040a98f0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 >> v1:172.21.15.47:6790/0 conn(0x7fad041430c0 legacy=0x7fad0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 >> v1:172.21.15.47:6789/0 conn(0x7fad0413f710 legacy=0x7fad041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad040b44c0 con 0x7fad040a98f0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7facec001690 con 0x7fad040a98f0 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad027fc640 10 monclient: dump: 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad09725640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:42:50.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.207+0000 7fad09725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facec0017f0 con 0x7fad040a98f0 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad027fc640 1 -- 172.21.15.47:0/373800178 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7facec001c00 con 0x7fad040a98f0 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad09725640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad09725640 4 set_mon_vals no callback set 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.218+0000 7fad027fc640 10 monclient: dump: 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.219 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.220 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.220 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:50.220 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:50.220 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:50.220 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:50.220 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:50.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 10 monclient: shutdown 2024-03-25T19:42:50.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 1 -- 172.21.15.47:0/373800178 >> v1:172.21.15.142:6789/0 conn(0x7fad040a98f0 legacy=0x7fad040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 1 -- 172.21.15.47:0/373800178 shutdown_connections 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 1 -- 172.21.15.47:0/373800178 >> 172.21.15.47:0/373800178 conn(0x7fad040b7ea0 msgr2=0x7fad040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 1 -- 172.21.15.47:0/373800178 shutdown_connections 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.219+0000 7fad09725640 1 -- 172.21.15.47:0/373800178 wait complete. 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 10 monclient: build_initial_monmap 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 10 monclient: monmap: 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:50.222 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 1 librados: starting msgr at 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 1 librados: starting objecter 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 1 Processor -- start 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 1 -- start start 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 1 librados: setting wanted keys 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 1 librados: calling monclient init 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.220+0000 7fad09725640 10 monclient: init 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient(hunting): picked mon.noname-c con 0x7fad040a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient(hunting): picked mon.noname-a con 0x7fad0413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient(hunting): picked mon.noname-b con 0x7fad041430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:50.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad04138b10 con 0x7fad0413f710 2024-03-25T19:42:50.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad04136e10 con 0x7fad041430c0 2024-03-25T19:42:50.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad04138d60 con 0x7fad040a98f0 2024-03-25T19:42:50.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:50.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.221+0000 7fad09725640 10 monclient(hunting): authenticate will time out at 6085.508343s 2024-03-25T19:42:50.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.226+0000 7facfbfff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fad0413f710 0x7fad041df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35776/0 (socket says 172.21.15.47:35776) 2024-03-25T19:42:50.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.226+0000 7facfbfff640 1 -- 172.21.15.47:0/2386673776 learned_addr learned my addr 172.21.15.47:0/2386673776 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:50.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.229+0000 7facfb7fe640 10 client.?.objecter ms_handle_connect 0x7fad0413f710 2024-03-25T19:42:50.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.229+0000 7facfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:50.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.235+0000 7facfb7fe640 10 client.?.objecter ms_handle_connect 0x7fad041430c0 2024-03-25T19:42:50.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.235+0000 7facfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:50.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.242+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007509941 0 0) 0x7fad04138b10 con 0x7fad0413f710 2024-03-25T19:42:50.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.242+0000 7facfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.242+0000 7facfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.242+0000 7facfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.242+0000 7facfb7fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:42:50.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.242+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8004eb0 con 0x7fad0413f710 2024-03-25T19:42:50.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.244+0000 7facfb7fe640 10 client.?.objecter ms_handle_connect 0x7fad040a98f0 2024-03-25T19:42:50.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.244+0000 7facfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:50.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.252+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658933827 0 0) 0x7fad04136e10 con 0x7fad041430c0 2024-03-25T19:42:50.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.252+0000 7facfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:50.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.252+0000 7facfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:50.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.252+0000 7facfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:50.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.252+0000 7facfb7fe640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:42:50.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.252+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad04138b10 con 0x7fad041430c0 2024-03-25T19:42:50.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078165133 0 0) 0x7face8004eb0 con 0x7fad0413f710 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad04136e10 con 0x7fad0413f710 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7facf40019d0 con 0x7fad0413f710 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:50.255 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:50.256 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:50.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.254+0000 7facfb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619582687 0 0) 0x7fad04136e10 con 0x7fad0413f710 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 >> v1:172.21.15.47:6790/0 conn(0x7fad040a98f0 legacy=0x7fad04137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 >> v1:172.21.15.142:6789/0 conn(0x7fad041430c0 legacy=0x7fad041e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 1 monclient: found mon.a 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:42:50.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04139040 con 0x7fad0413f710 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7facfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 5 monclient: authenticate success, global_id 4593 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 10 monclient: _renew_subs 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad04139380 con 0x7fad0413f710 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 10 monclient: _renew_subs 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad04139920 con 0x7fad0413f710 2024-03-25T19:42:50.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.263+0000 7fad09725640 1 librados: init done 2024-03-25T19:42:50.266 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.264+0000 7fad09725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:50.266 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.265+0000 7fad09725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:50.266 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.265+0000 7fad09725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:42:50.266 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.265+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad040627b0 con 0x7fad0413f710 2024-03-25T19:42:50.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facf40035f0 con 0x7fad0413f710 2024-03-25T19:42:50.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7facfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:50.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7facf4003980 con 0x7fad0413f710 2024-03-25T19:42:50.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7facfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:50.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7facfb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7facfb7fe640 10 monclient: dump: 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:50.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.615+0000 7fad01ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:50.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.616+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7facf4003ba0 con 0x7fad0413f710 2024-03-25T19:42:50.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.617+0000 7fad02ffd640 10 monclient: get_auth_request con 0x7face806dd10 auth_method 0 2024-03-25T19:42:50.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.617+0000 7facfb7fe640 10 client.4593.objecter ms_handle_connect 0x7face806dd10 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7facf4054780 con 0x7fad0413f710 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 10 client.4593.objecter ms_dispatch 0x7fad04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facf4050290 con 0x7fad0413f710 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:50.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.623+0000 7facfb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:50.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.863+0000 7fad09725640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:50.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.863+0000 7fad09725640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:50.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.863+0000 7fad09725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:42:50.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.863+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad041391d0 con 0x7fad0413f710 2024-03-25T19:42:50.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.957+0000 7facfb7fe640 1 -- 172.21.15.47:0/2386673776 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7facf40521c0 con 0x7fad0413f710 2024-03-25T19:42:50.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.957+0000 7facfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:50.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.957+0000 7facfb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:50.958 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:50.958 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 10 librados: watch_flush enter 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 10 librados: watch_flush exit 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 10 client.4593.objecter successfully canceled tick 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 >> v1:172.21.15.142:6800/1025302268 conn(0x7face806dd10 legacy=0x7face80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 10 monclient: shutdown 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 >> v1:172.21.15.47:6789/0 conn(0x7fad0413f710 legacy=0x7fad041df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 shutdown_connections 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 >> 172.21.15.47:0/2386673776 conn(0x7fad040b7ea0 msgr2=0x7fad04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 shutdown_connections 2024-03-25T19:42:50.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.959+0000 7fad09725640 1 -- 172.21.15.47:0/2386673776 wait complete. 2024-03-25T19:42:50.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:50.960+0000 7fad09725640 1 librados: shutdown 2024-03-25T19:42:50.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:57.778 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:57.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.896+0000 7fefb390e640 10 monclient: get_monmap_and_config 2024-03-25T19:42:57.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.896+0000 7fefb390e640 10 monclient: build_initial_monmap 2024-03-25T19:42:57.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.896+0000 7fefb390e640 10 monclient: monmap: 2024-03-25T19:42:57.897 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:57.897 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:57.897 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:42:57.898530+0000 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:42:57.898530+0000 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:42:57.898 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:57.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.897+0000 7fefb390e640 1 Processor -- start 2024-03-25T19:42:57.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.898+0000 7fefb390e640 1 -- start start 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.898+0000 7fefb390e640 10 monclient: init 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.898+0000 7fefb390e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.898+0000 7fefb390e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 10 monclient(hunting): picked mon.noname-b con 0x7fefac0705b0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 10 monclient(hunting): picked mon.noname-c con 0x7fefac1402d0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 10 monclient(hunting): picked mon.noname-a con 0x7fefac149ae0 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0718c0 con 0x7fefac149ae0 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0714c0 con 0x7fefac1402d0 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac072b50 con 0x7fefac0705b0 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:57.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.899+0000 7fefb390e640 10 monclient(hunting): authenticate will time out at 6093.186042s 2024-03-25T19:42:57.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.906+0000 7fefb0ea9640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fefac1402d0 0x7fefac140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:36808/0 (socket says 172.21.15.47:36808) 2024-03-25T19:42:57.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.906+0000 7fefb0ea9640 1 -- 172.21.15.47:0/3709839907 learned_addr learned my addr 172.21.15.47:0/3709839907 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:57.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.915+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246250670 0 0) 0x7fefac0714c0 con 0x7fefac1402d0 2024-03-25T19:42:57.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.915+0000 7fefa3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:57.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.915+0000 7fefa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:57.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.915+0000 7fefa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:57.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.915+0000 7fefa3fff640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:42:57.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.915+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef98003520 con 0x7fefac1402d0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4176829185 0 0) 0x7fef98003520 con 0x7fefac1402d0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 >> v1:172.21.15.47:6790/0 conn(0x7fefac0705b0 legacy=0x7fefac13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 >> v1:172.21.15.47:6789/0 conn(0x7fefac149ae0 legacy=0x7fefac14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 1 monclient: found mon.noname-c 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac13dd10 con 0x7fefac1402d0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fef9c001690 con 0x7fefac1402d0 2024-03-25T19:42:57.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefa3fff640 10 monclient: dump: 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:57.924 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:57.925 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:57.925 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:57.925 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:57.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefb390e640 5 monclient: authenticate success, global_id 4594 2024-03-25T19:42:57.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.922+0000 7fefb390e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.935+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef9c001860 con 0x7fefac1402d0 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.935+0000 7fefa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.935+0000 7fefa3fff640 1 -- 172.21.15.47:0/3709839907 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fef9c001cb0 con 0x7fefac1402d0 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.935+0000 7fefa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.935+0000 7fefa3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.935+0000 7fefa3fff640 10 monclient: dump: 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:57.936 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.936+0000 7fefb390e640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:57.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.936+0000 7fefb390e640 4 set_mon_vals no callback set 2024-03-25T19:42:57.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.936+0000 7fefb390e640 10 monclient: shutdown 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.936+0000 7fefb390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.936+0000 7fefb390e640 1 -- 172.21.15.47:0/3709839907 >> v1:172.21.15.142:6789/0 conn(0x7fefac1402d0 legacy=0x7fefac140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.937+0000 7fefb390e640 1 -- 172.21.15.47:0/3709839907 shutdown_connections 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.937+0000 7fefb390e640 1 -- 172.21.15.47:0/3709839907 >> 172.21.15.47:0/3709839907 conn(0x7fefac0b5d10 msgr2=0x7fefac0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.937+0000 7fefb390e640 1 -- 172.21.15.47:0/3709839907 shutdown_connections 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.937+0000 7fefb390e640 1 -- 172.21.15.47:0/3709839907 wait complete. 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.937+0000 7fefb390e640 10 monclient: build_initial_monmap 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.937+0000 7fefb390e640 10 monclient: monmap: 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:42:57.939 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 1 librados: starting msgr at 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 1 librados: starting objecter 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 1 Processor -- start 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 1 -- start start 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 1 librados: setting wanted keys 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 1 librados: calling monclient init 2024-03-25T19:42:57.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.938+0000 7fefb390e640 10 monclient: init 2024-03-25T19:42:57.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:57.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:57.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient(hunting): picked mon.noname-b con 0x7fefac0705b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:42:57.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient(hunting): picked mon.noname-a con 0x7fefac1402d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:42:57.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient(hunting): picked mon.noname-c con 0x7fefac149ae0 addr v1:172.21.15.47:6790/0 2024-03-25T19:42:57.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac13eca0 con 0x7fefac1402d0 2024-03-25T19:42:57.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac13e950 con 0x7fefac0705b0 2024-03-25T19:42:57.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac06d170 con 0x7fefac149ae0 2024-03-25T19:42:57.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:57.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.939+0000 7fefb390e640 10 monclient(hunting): authenticate will time out at 6093.226281s 2024-03-25T19:42:57.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.948+0000 7fefb0ea9640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fefac1402d0 0x7fefac06c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42914/0 (socket says 172.21.15.47:42914) 2024-03-25T19:42:57.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.949+0000 7fefb0ea9640 1 -- 172.21.15.47:0/377709156 learned_addr learned my addr 172.21.15.47:0/377709156 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:42:57.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.950+0000 7fefa1ffb640 10 client.?.objecter ms_handle_connect 0x7fefac1402d0 2024-03-25T19:42:57.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.950+0000 7fefa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:57.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.952+0000 7fefa1ffb640 10 client.?.objecter ms_handle_connect 0x7fefac149ae0 2024-03-25T19:42:57.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.952+0000 7fefa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:57.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.956+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852874253 0 0) 0x7fefac13eca0 con 0x7fefac1402d0 2024-03-25T19:42:57.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.956+0000 7fefa1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:57.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.956+0000 7fefa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:57.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.956+0000 7fefa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:57.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.956+0000 7fefa1ffb640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:42:57.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.956+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef90004eb0 con 0x7fefac1402d0 2024-03-25T19:42:57.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.959+0000 7fefa1ffb640 10 client.?.objecter ms_handle_connect 0x7fefac0705b0 2024-03-25T19:42:57.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.959+0000 7fefa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:57.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.964+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036904318 0 0) 0x7fefac06d170 con 0x7fefac149ae0 2024-03-25T19:42:57.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.964+0000 7fefa1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:57.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.964+0000 7fefa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:57.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.964+0000 7fefa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:57.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.964+0000 7fefa1ffb640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:42:57.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.964+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac13eca0 con 0x7fefac149ae0 2024-03-25T19:42:57.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706067585 0 0) 0x7fefac13e950 con 0x7fefac0705b0 2024-03-25T19:42:57.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:57.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:57.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:57.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac06d170 con 0x7fefac0705b0 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 489507755 0 0) 0x7fef90004eb0 con 0x7fefac1402d0 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefac13e950 con 0x7fefac1402d0 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fef9c001b70 con 0x7fefac1402d0 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:57.976 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:57.977 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:57.977 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:57.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.974+0000 7fefa1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:57.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.976+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63559421 0 0) 0x7fefac13eca0 con 0x7fefac149ae0 2024-03-25T19:42:57.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.976+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef90004eb0 con 0x7fefac149ae0 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.976+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fefa4001a70 con 0x7fefac149ae0 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.976+0000 7fefa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.976+0000 7fefa1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.976+0000 7fefa1ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:57.978 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:57.979 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:57.979 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:57.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2181026935 0 0) 0x7fef90004eb0 con 0x7fefac149ae0 2024-03-25T19:42:57.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 >> v1:172.21.15.142:6789/0 conn(0x7fefac0705b0 legacy=0x7fefac132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 >> v1:172.21.15.47:6789/0 conn(0x7fefac1402d0 legacy=0x7fefac06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 1 monclient: found mon.c 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac134920 con 0x7fefac149ae0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefb390e640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefb390e640 10 monclient: _renew_subs 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefb390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.983+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac133460 con 0x7fefac149ae0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.984+0000 7fefb390e640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.984+0000 7fefb390e640 10 monclient: _renew_subs 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.984+0000 7fefb390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:57.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.984+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac13e1b0 con 0x7fefac149ae0 2024-03-25T19:42:57.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.984+0000 7fefb390e640 1 librados: init done 2024-03-25T19:42:57.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.985+0000 7fefb390e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:57.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.985+0000 7fefb390e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:57.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.985+0000 7fefb390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:57.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:57.985+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac13eae0 con 0x7fefac149ae0 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa4001f60 con 0x7fefac149ae0 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fefa4003b10 con 0x7fefac149ae0 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa1ffb640 10 monclient: dump: 2024-03-25T19:42:58.307 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.306+0000 7fefa37fe640 4 set_mon_vals no callback set 2024-03-25T19:42:58.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.307+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fefa4019ae0 con 0x7fefac149ae0 2024-03-25T19:42:58.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.307+0000 7fefb16aa640 10 monclient: get_auth_request con 0x7fef9006df60 auth_method 0 2024-03-25T19:42:58.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.307+0000 7fefa1ffb640 10 client.4586.objecter ms_handle_connect 0x7fef9006df60 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.312+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fefa4053730 con 0x7fefac149ae0 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.312+0000 7fefa1ffb640 10 client.4586.objecter ms_dispatch 0x7fefac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.312+0000 7fefa1ffb640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.312+0000 7fefa1ffb640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:58.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.313+0000 7fefa1ffb640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T19:42:58.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.313+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefa404f070 con 0x7fefac149ae0 2024-03-25T19:42:58.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.313+0000 7fefa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:58.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.313+0000 7fefa1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:58.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.570+0000 7fefb390e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:58.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.570+0000 7fefb390e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:58.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.570+0000 7fefb390e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:42:58.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.570+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefac0627b0 con 0x7fefac149ae0 2024-03-25T19:42:58.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.600+0000 7fefa1ffb640 1 -- 172.21.15.47:0/377709156 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fefa4051170 con 0x7fefac149ae0 2024-03-25T19:42:58.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.600+0000 7fefa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:58.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.600+0000 7fefa1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:58.601 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:42:58.602 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:58.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.602+0000 7fefb390e640 10 librados: watch_flush enter 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 10 librados: watch_flush exit 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 10 client.4586.objecter successfully canceled tick 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 >> v1:172.21.15.142:6800/1025302268 conn(0x7fef9006df60 legacy=0x7fef90070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 10 monclient: shutdown 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 >> v1:172.21.15.47:6790/0 conn(0x7fefac149ae0 legacy=0x7fefac06ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 shutdown_connections 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 >> 172.21.15.47:0/377709156 conn(0x7fefac0b5d10 msgr2=0x7fefac13b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 shutdown_connections 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 -- 172.21.15.47:0/377709156 wait complete. 2024-03-25T19:42:58.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:42:58.603+0000 7fefb390e640 1 librados: shutdown 2024-03-25T19:42:58.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:03.692 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:40.927 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.698 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:40.933 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.734 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:40.970 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.747 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:40.937 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.754 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:40.989 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.761 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:40.951 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.768 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:41.004 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.784 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:41.019 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:03.787 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:08:40.977 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.420 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:05.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.541+0000 7f6dfea63640 10 monclient: get_monmap_and_config 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.541+0000 7f6dfea63640 10 monclient: build_initial_monmap 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.541+0000 7f6dfea63640 10 monclient: monmap: 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:05.542170+0000 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:05.542170+0000 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:05.541 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.542+0000 7f6dfea63640 1 Processor -- start 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.542+0000 7f6dfea63640 1 -- start start 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.542+0000 7f6dfea63640 10 monclient: init 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient(hunting): picked mon.noname-b con 0x7f6df80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient(hunting): picked mon.noname-a con 0x7f6df813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient(hunting): picked mon.noname-c con 0x7f6df81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df8142520 con 0x7f6df813f710 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80b28a0 con 0x7f6df81430c0 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80b2a30 con 0x7f6df80a98f0 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:05.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.543+0000 7f6dfea63640 10 monclient(hunting): authenticate will time out at 6100.829340s 2024-03-25T19:43:05.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.559+0000 7f6df7fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6df80a98f0 0x7f6df80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55218/0 (socket says 172.21.15.47:55218) 2024-03-25T19:43:05.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.559+0000 7f6df7fff640 1 -- 172.21.15.47:0/3313444538 learned_addr learned my addr 172.21.15.47:0/3313444538 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:05.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.579+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497202748 0 0) 0x7f6df8142520 con 0x7f6df813f710 2024-03-25T19:43:05.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.579+0000 7f6df6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.579+0000 7f6df6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.579+0000 7f6df6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.579+0000 7f6df6ffd640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:43:05.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.579+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de0003520 con 0x7f6df813f710 2024-03-25T19:43:05.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.583+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676011209 0 0) 0x7f6df80b2a30 con 0x7f6df80a98f0 2024-03-25T19:43:05.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.583+0000 7f6df6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.583+0000 7f6df6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.583+0000 7f6df6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.583+0000 7f6df6ffd640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.583+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df8142520 con 0x7f6df80a98f0 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.584+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753190021 0 0) 0x7f6df80b28a0 con 0x7f6df81430c0 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.584+0000 7f6df6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.584+0000 7f6df6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.584+0000 7f6df6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.584+0000 7f6df6ffd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:43:05.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.584+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df80b2a30 con 0x7f6df81430c0 2024-03-25T19:43:05.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3798825609 0 0) 0x7f6de0003520 con 0x7f6df813f710 2024-03-25T19:43:05.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 >> v1:172.21.15.47:6790/0 conn(0x7f6df80a98f0 legacy=0x7f6df80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 >> v1:172.21.15.142:6789/0 conn(0x7f6df81430c0 legacy=0x7f6df814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:05.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80b44c0 con 0x7f6df813f710 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6dec001690 con 0x7f6df813f710 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6dfea63640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6dfea63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.594+0000 7f6df6ffd640 10 monclient: dump: 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:05.595 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.595+0000 7f6dfea63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dec0017f0 con 0x7f6df813f710 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6df6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6df6ffd640 1 -- 172.21.15.47:0/3313444538 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6dec001c00 con 0x7f6df813f710 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6df6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6df6ffd640 10 monclient: dump: 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:05.620 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:05.621 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:05.621 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:05.621 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6dfea63640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:05.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.620+0000 7f6dfea63640 4 set_mon_vals no callback set 2024-03-25T19:43:05.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 10 monclient: shutdown 2024-03-25T19:43:05.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:05.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 1 -- 172.21.15.47:0/3313444538 >> v1:172.21.15.47:6789/0 conn(0x7f6df813f710 legacy=0x7f6df81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 1 -- 172.21.15.47:0/3313444538 shutdown_connections 2024-03-25T19:43:05.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 1 -- 172.21.15.47:0/3313444538 >> 172.21.15.47:0/3313444538 conn(0x7f6df80b7ea0 msgr2=0x7f6df80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:05.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 1 -- 172.21.15.47:0/3313444538 shutdown_connections 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.621+0000 7f6dfea63640 1 -- 172.21.15.47:0/3313444538 wait complete. 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.622+0000 7f6dfea63640 10 monclient: build_initial_monmap 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.622+0000 7f6dfea63640 10 monclient: monmap: 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.622+0000 7f6dfea63640 1 librados: starting msgr at 2024-03-25T19:43:05.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.622+0000 7f6dfea63640 1 librados: starting objecter 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.622+0000 7f6dfea63640 1 Processor -- start 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 1 -- start start 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 1 librados: setting wanted keys 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 1 librados: calling monclient init 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient: init 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient(hunting): picked mon.noname-b con 0x7f6df80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient(hunting): picked mon.noname-a con 0x7f6df813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient(hunting): picked mon.noname-c con 0x7f6df81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df81d7d80 con 0x7f6df813f710 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df81d7f10 con 0x7f6df80a98f0 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df81d80a0 con 0x7f6df81430c0 2024-03-25T19:43:05.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:05.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.623+0000 7f6dfea63640 10 monclient(hunting): authenticate will time out at 6100.909869s 2024-03-25T19:43:05.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.629+0000 7f6dfd000640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6df81430c0 0x7f6df8135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55294/0 (socket says 172.21.15.47:55294) 2024-03-25T19:43:05.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.629+0000 7f6dfd000640 1 -- 172.21.15.47:0/2956619378 learned_addr learned my addr 172.21.15.47:0/2956619378 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:05.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.635+0000 7f6df4ff9640 10 client.?.objecter ms_handle_connect 0x7f6df81430c0 2024-03-25T19:43:05.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.635+0000 7f6df4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:05.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.635+0000 7f6df4ff9640 10 client.?.objecter ms_handle_connect 0x7f6df80a98f0 2024-03-25T19:43:05.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.635+0000 7f6df4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:05.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.640+0000 7f6df4ff9640 10 client.?.objecter ms_handle_connect 0x7f6df813f710 2024-03-25T19:43:05.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.640+0000 7f6df4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.643+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124299950 0 0) 0x7f6df80b2a30 con 0x7f6df81430c0 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.643+0000 7f6df4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.643+0000 7f6df4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.643+0000 7f6df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.643+0000 7f6df4ff9640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.643+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de4004eb0 con 0x7f6df81430c0 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.645+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466181619 0 0) 0x7f6df81d7f10 con 0x7f6df80a98f0 2024-03-25T19:43:05.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.645+0000 7f6df4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:05.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.645+0000 7f6df4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:05.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.645+0000 7f6df4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:05.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.645+0000 7f6df4ff9640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:43:05.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.645+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df80b2a30 con 0x7f6df80a98f0 2024-03-25T19:43:05.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.652+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367324909 0 0) 0x7f6df80b2a30 con 0x7f6df80a98f0 2024-03-25T19:43:05.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.652+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df81d7f10 con 0x7f6df80a98f0 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655706976 0 0) 0x7f6de4004eb0 con 0x7f6df81430c0 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df80b2a30 con 0x7f6df81430c0 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6df0001ad0 con 0x7f6df81430c0 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.653 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.653+0000 7f6df4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.654+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6de80019b0 con 0x7f6df80a98f0 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.654+0000 7f6df4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.654+0000 7f6df4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.654+0000 7f6df4ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:05.654 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:05.655 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2625794790 0 0) 0x7f6df81d7f10 con 0x7f6df80a98f0 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 >> v1:172.21.15.47:6790/0 conn(0x7f6df81430c0 legacy=0x7f6df8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 >> v1:172.21.15.47:6789/0 conn(0x7f6df813f710 legacy=0x7f6df8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 1 monclient: found mon.b 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80b7200 con 0x7f6df80a98f0 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.661+0000 7f6df4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:05.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 10 monclient: _renew_subs 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df80b7450 con 0x7f6df80a98f0 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 10 monclient: _renew_subs 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df81e3230 con 0x7f6df80a98f0 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.662+0000 7f6dfea63640 1 librados: init done 2024-03-25T19:43:05.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.663+0000 7f6dfea63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:05.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.663+0000 7f6dfea63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:05.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.663+0000 7f6dfea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:05.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.663+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df81d84a0 con 0x7f6df80a98f0 2024-03-25T19:43:05.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.942+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6de80047e0 con 0x7f6df80a98f0 2024-03-25T19:43:05.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.942+0000 7f6df4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:05.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.943+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f6de8004af0 con 0x7f6df80a98f0 2024-03-25T19:43:05.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.943+0000 7f6df4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.943+0000 7f6df4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.943+0000 7f6df4ff9640 10 monclient: dump: 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:05.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.943+0000 7f6df67fc640 4 set_mon_vals no callback set 2024-03-25T19:43:05.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.943+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f6de801ab80 con 0x7f6df80a98f0 2024-03-25T19:43:05.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.944+0000 7f6df77fe640 10 monclient: get_auth_request con 0x7f6de406def0 auth_method 0 2024-03-25T19:43:05.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.944+0000 7f6df4ff9640 10 client.4603.objecter ms_handle_connect 0x7f6de406def0 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f6de8054800 con 0x7f6df80a98f0 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 10 client.4603.objecter ms_dispatch 0x7f6df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:05.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T19:43:05.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6de8050140 con 0x7f6df80a98f0 2024-03-25T19:43:05.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:05.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:05.948+0000 7f6df4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:06.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.208+0000 7f6dfea63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:06.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.208+0000 7f6dfea63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:06.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.208+0000 7f6dfea63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:06.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.208+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df8074cc0 con 0x7f6df80a98f0 2024-03-25T19:43:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.233+0000 7f6df4ff9640 1 -- 172.21.15.47:0/2956619378 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f6de8052240 con 0x7f6df80a98f0 2024-03-25T19:43:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.233+0000 7f6df4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.233+0000 7f6df4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:06.233 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:06.233 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.235+0000 7f6dfea63640 10 librados: watch_flush enter 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 10 librados: watch_flush exit 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 10 client.4603.objecter successfully canceled tick 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 >> v1:172.21.15.142:6800/1025302268 conn(0x7f6de406def0 legacy=0x7f6de40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 10 monclient: shutdown 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 >> v1:172.21.15.142:6789/0 conn(0x7f6df80a98f0 legacy=0x7f6df81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 shutdown_connections 2024-03-25T19:43:06.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 >> 172.21.15.47:0/2956619378 conn(0x7f6df80b7ea0 msgr2=0x7f6df8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:06.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 shutdown_connections 2024-03-25T19:43:06.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 -- 172.21.15.47:0/2956619378 wait complete. 2024-03-25T19:43:06.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:06.236+0000 7f6dfea63640 1 librados: shutdown 2024-03-25T19:43:06.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:13.054 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:13.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.172+0000 7f09f062a640 10 monclient: get_monmap_and_config 2024-03-25T19:43:13.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.172+0000 7f09f062a640 10 monclient: build_initial_monmap 2024-03-25T19:43:13.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.172+0000 7f09f062a640 10 monclient: monmap: 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:13.172852+0000 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:13.172852+0000 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:13.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.173+0000 7f09f062a640 1 Processor -- start 2024-03-25T19:43:13.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.173+0000 7f09f062a640 1 -- start start 2024-03-25T19:43:13.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.173+0000 7f09f062a640 10 monclient: init 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient(hunting): picked mon.noname-b con 0x7f09e806ea40 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient(hunting): picked mon.noname-a con 0x7f09e8071800 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient(hunting): picked mon.noname-c con 0x7f09e8071c40 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e806f1f0 con 0x7f09e8071800 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e8070d00 con 0x7f09e8071c40 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e80720b0 con 0x7f09e806ea40 2024-03-25T19:43:13.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:13.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.174+0000 7f09f062a640 10 monclient(hunting): authenticate will time out at 6108.460615s 2024-03-25T19:43:13.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.190+0000 7f09edbc5640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f09e8071800 0x7f09e8147070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53042/0 (socket says 172.21.15.47:53042) 2024-03-25T19:43:13.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.190+0000 7f09edbc5640 1 -- 172.21.15.47:0/403466420 learned_addr learned my addr 172.21.15.47:0/403466420 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.206+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016874773 0 0) 0x7f09e8070d00 con 0x7f09e8071c40 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.206+0000 7f09ed3c4640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.206+0000 7f09ed3c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.206+0000 7f09ed3c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.206+0000 7f09ed3c4640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.206+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d4005730 con 0x7f09e8071c40 2024-03-25T19:43:13.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.207+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651445849 0 0) 0x7f09e80720b0 con 0x7f09e806ea40 2024-03-25T19:43:13.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.207+0000 7f09ed3c4640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.207+0000 7f09ed3c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.207+0000 7f09ed3c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.207+0000 7f09ed3c4640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:43:13.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.207+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8070d00 con 0x7f09e806ea40 2024-03-25T19:43:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.215+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872720210 0 0) 0x7f09e806f1f0 con 0x7f09e8071800 2024-03-25T19:43:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.215+0000 7f09ed3c4640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.215+0000 7f09ed3c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.215+0000 7f09ed3c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.215+0000 7f09ed3c4640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:43:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.215+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e80720b0 con 0x7f09e8071800 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 413407392 0 0) 0x7f09d4005730 con 0x7f09e8071c40 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 >> v1:172.21.15.47:6790/0 conn(0x7f09e806ea40 legacy=0x7f09e813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 >> v1:172.21.15.47:6789/0 conn(0x7f09e8071800 legacy=0x7f09e8147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: _finish_hunting 0 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 1 monclient: found mon.noname-c 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e8072780 con 0x7f09e8071c40 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: _finish_auth 0 2024-03-25T19:43:13.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: _check_auth_tickets 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f09e0001a10 con 0x7f09e8071c40 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09ed3c4640 10 monclient: dump: 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:13.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09f062a640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:43:13.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.238+0000 7f09f062a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:13.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09e0001b70 con 0x7f09e8071c40 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09ed3c4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09ed3c4640 1 -- 172.21.15.47:0/403466420 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f09e0001f80 con 0x7f09e8071c40 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09ed3c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09ed3c4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09ed3c4640 10 monclient: dump: 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:13.258 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:13.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09f062a640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:13.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.258+0000 7f09f062a640 4 set_mon_vals no callback set 2024-03-25T19:43:13.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 10 monclient: shutdown 2024-03-25T19:43:13.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:13.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 1 -- 172.21.15.47:0/403466420 >> v1:172.21.15.142:6789/0 conn(0x7f09e8071c40 legacy=0x7f09e814a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 1 -- 172.21.15.47:0/403466420 shutdown_connections 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 1 -- 172.21.15.47:0/403466420 >> 172.21.15.47:0/403466420 conn(0x7f09e806b630 msgr2=0x7f09e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 1 -- 172.21.15.47:0/403466420 shutdown_connections 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.259+0000 7f09f062a640 1 -- 172.21.15.47:0/403466420 wait complete. 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.260+0000 7f09f062a640 10 monclient: build_initial_monmap 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.260+0000 7f09f062a640 10 monclient: monmap: 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:13.260 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:13.261 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 1 librados: starting msgr at 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 1 librados: starting objecter 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 1 Processor -- start 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 1 -- start start 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 1 librados: setting wanted keys 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 1 librados: calling monclient init 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.261+0000 7f09f062a640 10 monclient: init 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:13.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient(hunting): picked mon.noname-b con 0x7f09e806ea40 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient(hunting): picked mon.noname-c con 0x7f09e8071800 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient(hunting): picked mon.noname-a con 0x7f09e8071c40 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e80ac3e0 con 0x7f09e8071c40 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e80ac6c0 con 0x7f09e806ea40 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e80aae10 con 0x7f09e8071800 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:13.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.262+0000 7f09f062a640 10 monclient(hunting): authenticate will time out at 6108.548475s 2024-03-25T19:43:13.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.274+0000 7f09edbc5640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f09e8071800 0x7f09e8134ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39136/0 (socket says 172.21.15.47:39136) 2024-03-25T19:43:13.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.274+0000 7f09edbc5640 1 -- 172.21.15.47:0/4176634013 learned_addr learned my addr 172.21.15.47:0/4176634013 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:13.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.276+0000 7f09deffd640 10 client.?.objecter ms_handle_connect 0x7f09e8071c40 2024-03-25T19:43:13.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.276+0000 7f09deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:13.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.280+0000 7f09deffd640 10 client.?.objecter ms_handle_connect 0x7f09e8071800 2024-03-25T19:43:13.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.280+0000 7f09deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:13.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.284+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373477448 0 0) 0x7f09e80aae10 con 0x7f09e8071800 2024-03-25T19:43:13.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.284+0000 7f09deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.284+0000 7f09deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.284+0000 7f09deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.284+0000 7f09deffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:43:13.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.284+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc004eb0 con 0x7f09e8071800 2024-03-25T19:43:13.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.292+0000 7f09deffd640 10 client.?.objecter ms_handle_connect 0x7f09e806ea40 2024-03-25T19:43:13.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.292+0000 7f09deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:13.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.300+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135194027 0 0) 0x7f09e80ac3e0 con 0x7f09e8071c40 2024-03-25T19:43:13.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:13.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:13.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:13.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:43:13.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e80aae10 con 0x7f09e8071c40 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139154013 0 0) 0x7f09cc004eb0 con 0x7f09e8071800 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e80ac3e0 con 0x7f09e8071800 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f09e4001910 con 0x7f09e8071800 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): dump: 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:13.301 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:13.302 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:13.302 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:13.302 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:13.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.301+0000 7f09deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:13.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.307+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1617011368 0 0) 0x7f09e80ac3e0 con 0x7f09e8071800 2024-03-25T19:43:13.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.307+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 >> v1:172.21.15.142:6789/0 conn(0x7f09e806ea40 legacy=0x7f09e81d30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 >> v1:172.21.15.47:6789/0 conn(0x7f09e8071c40 legacy=0x7f09e81de920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 1 monclient: found mon.c 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e80ab330 con 0x7f09e8071800 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 5 monclient: authenticate success, global_id 4598 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 10 monclient: _renew_subs 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e81d37f0 con 0x7f09e8071800 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 10 monclient: _renew_subs 2024-03-25T19:43:13.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:13.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e80ad030 con 0x7f09e8071800 2024-03-25T19:43:13.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.308+0000 7f09f062a640 1 librados: init done 2024-03-25T19:43:13.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.309+0000 7f09f062a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:13.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.309+0000 7f09f062a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:13.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.309+0000 7f09f062a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:13.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.309+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e80ab520 con 0x7f09e8071800 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09e40047e0 con 0x7f09e8071800 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f09e4004bf0 con 0x7f09e8071800 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09deffd640 10 monclient: dump: 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:13.613 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:13.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.613+0000 7f09ecbc3640 4 set_mon_vals no callback set 2024-03-25T19:43:13.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.614+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f09e4004e10 con 0x7f09e8071800 2024-03-25T19:43:13.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.615+0000 7f09ee3c6640 10 monclient: get_auth_request con 0x7f09cc06dd90 auth_method 0 2024-03-25T19:43:13.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.615+0000 7f09deffd640 10 client.4598.objecter ms_handle_connect 0x7f09cc06dd90 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.621+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f09e40548b0 con 0x7f09e8071800 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.621+0000 7f09deffd640 10 client.4598.objecter ms_dispatch 0x7f09e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.621+0000 7f09deffd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.621+0000 7f09deffd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.621+0000 7f09deffd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.621+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09e40503c0 con 0x7f09e8071800 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.622+0000 7f09deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:13.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.622+0000 7f09deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.858+0000 7f09f062a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.858+0000 7f09f062a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.858+0000 7f09f062a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:13.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.858+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09e8074d80 con 0x7f09e8071800 2024-03-25T19:43:13.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.885+0000 7f09deffd640 1 -- 172.21.15.47:0/4176634013 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f09e8074d80 con 0x7f09e8071800 2024-03-25T19:43:13.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.885+0000 7f09deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:13.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.885+0000 7f09deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:13.885 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:13.885 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:13.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 10 librados: watch_flush enter 2024-03-25T19:43:13.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 10 librados: watch_flush exit 2024-03-25T19:43:13.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 10 client.4598.objecter successfully canceled tick 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 >> v1:172.21.15.142:6800/1025302268 conn(0x7f09cc06dd90 legacy=0x7f09cc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 10 monclient: shutdown 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.887+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 >> v1:172.21.15.47:6790/0 conn(0x7f09e8071800 legacy=0x7f09e8134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.888+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 shutdown_connections 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.888+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 >> 172.21.15.47:0/4176634013 conn(0x7f09e806b630 msgr2=0x7f09e80aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.888+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 shutdown_connections 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.888+0000 7f09f062a640 1 -- 172.21.15.47:0/4176634013 wait complete. 2024-03-25T19:43:13.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:13.888+0000 7f09f062a640 1 librados: shutdown 2024-03-25T19:43:13.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:20.705 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:20.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.811+0000 7fd81434b640 10 monclient: get_monmap_and_config 2024-03-25T19:43:20.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.811+0000 7fd81434b640 10 monclient: build_initial_monmap 2024-03-25T19:43:20.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.811+0000 7fd81434b640 10 monclient: monmap: 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:20.812212+0000 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:20.812212+0000 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:20.811 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:20.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 1 Processor -- start 2024-03-25T19:43:20.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 1 -- start start 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 10 monclient: init 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 10 monclient(hunting): picked mon.noname-c con 0x7fd80c0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.812+0000 7fd81434b640 10 monclient(hunting): picked mon.noname-a con 0x7fd80c0b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.813+0000 7fd81434b640 10 monclient(hunting): picked mon.noname-b con 0x7fd80c14be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.813+0000 7fd81434b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c0af940 con 0x7fd80c0b1730 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.813+0000 7fd81434b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c0b0a70 con 0x7fd80c0b4d60 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.813+0000 7fd81434b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c0b0d50 con 0x7fd80c14be90 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.813+0000 7fd81434b640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:20.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.813+0000 7fd81434b640 10 monclient(hunting): authenticate will time out at 6116.099068s 2024-03-25T19:43:20.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.822+0000 7fd8118e6640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd80c0b1730 0x7fd80c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56662/0 (socket says 172.21.15.47:56662) 2024-03-25T19:43:20.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.822+0000 7fd8118e6640 1 -- 172.21.15.47:0/569178383 learned_addr learned my addr 172.21.15.47:0/569178383 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:20.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.836+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250177105 0 0) 0x7fd80c0b0a70 con 0x7fd80c0b4d60 2024-03-25T19:43:20.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.836+0000 7fd8110e5640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.836+0000 7fd8110e5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.836+0000 7fd8110e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.836+0000 7fd8110e5640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:43:20.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.836+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f8003520 con 0x7fd80c0b4d60 2024-03-25T19:43:20.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.844+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336881244 0 0) 0x7fd80c0af940 con 0x7fd80c0b1730 2024-03-25T19:43:20.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.844+0000 7fd8110e5640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.844+0000 7fd8110e5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.844+0000 7fd8110e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.844+0000 7fd8110e5640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:43:20.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.844+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd80c0b0a70 con 0x7fd80c0b1730 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860801511 0 0) 0x7fd7f8003520 con 0x7fd80c0b4d60 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 >> v1:172.21.15.47:6790/0 conn(0x7fd80c14be90 legacy=0x7fd80c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 >> v1:172.21.15.47:6789/0 conn(0x7fd80c0b1730 legacy=0x7fd80c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: _finish_hunting 0 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 1 monclient: found mon.noname-c 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c0b1030 con 0x7fd80c0b4d60 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: _finish_auth 0 2024-03-25T19:43:20.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: _check_auth_tickets 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd808001690 con 0x7fd80c0b4d60 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd8110e5640 10 monclient: dump: 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:20.851 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:20.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd81434b640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:43:20.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.850+0000 7fd81434b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8080017f0 con 0x7fd80c0b4d60 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd8110e5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd8110e5640 1 -- 172.21.15.47:0/569178383 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd808001c00 con 0x7fd80c0b4d60 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd8110e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd8110e5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd8110e5640 10 monclient: dump: 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:20.860 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:20.861 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:20.861 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:20.861 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:20.861 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:20.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd81434b640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:20.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.860+0000 7fd81434b640 4 set_mon_vals no callback set 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.861+0000 7fd81434b640 10 monclient: shutdown 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.861+0000 7fd81434b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.861+0000 7fd81434b640 1 -- 172.21.15.47:0/569178383 >> v1:172.21.15.142:6789/0 conn(0x7fd80c0b4d60 legacy=0x7fd80c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.862+0000 7fd81434b640 1 -- 172.21.15.47:0/569178383 shutdown_connections 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.862+0000 7fd81434b640 1 -- 172.21.15.47:0/569178383 >> 172.21.15.47:0/569178383 conn(0x7fd80c0617a0 msgr2=0x7fd80c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.862+0000 7fd81434b640 1 -- 172.21.15.47:0/569178383 shutdown_connections 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.862+0000 7fd81434b640 1 -- 172.21.15.47:0/569178383 wait complete. 2024-03-25T19:43:20.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 10 monclient: build_initial_monmap 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 10 monclient: monmap: 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:20.863 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:20.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 1 librados: starting msgr at 2024-03-25T19:43:20.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 1 librados: starting objecter 2024-03-25T19:43:20.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 1 Processor -- start 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 1 -- start start 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 1 librados: setting wanted keys 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 1 librados: calling monclient init 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.863+0000 7fd81434b640 10 monclient: init 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient(hunting): picked mon.noname-a con 0x7fd80c0b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient(hunting): picked mon.noname-b con 0x7fd80c0b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient(hunting): picked mon.noname-c con 0x7fd80c14be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c13a8d0 con 0x7fd80c0b1730 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c1d7840 con 0x7fd80c0b4d60 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c1d8470 con 0x7fd80c14be90 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:20.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.864+0000 7fd81434b640 10 monclient(hunting): authenticate will time out at 6116.150480s 2024-03-25T19:43:20.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.876+0000 7fd8118e6640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd80c0b4d60 0x7fd80c1d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:47882/0 (socket says 172.21.15.47:47882) 2024-03-25T19:43:20.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.876+0000 7fd8118e6640 1 -- 172.21.15.47:0/371231948 learned_addr learned my addr 172.21.15.47:0/371231948 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:20.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.877+0000 7fd802ffd640 10 client.?.objecter ms_handle_connect 0x7fd80c0b4d60 2024-03-25T19:43:20.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.877+0000 7fd802ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:20.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.882+0000 7fd802ffd640 10 client.?.objecter ms_handle_connect 0x7fd80c14be90 2024-03-25T19:43:20.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.882+0000 7fd802ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.884+0000 7fd802ffd640 10 client.?.objecter ms_handle_connect 0x7fd80c0b1730 2024-03-25T19:43:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.884+0000 7fd802ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:20.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.893+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262732272 0 0) 0x7fd80c1d8470 con 0x7fd80c14be90 2024-03-25T19:43:20.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.893+0000 7fd802ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.893+0000 7fd802ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.893+0000 7fd802ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.893+0000 7fd802ffd640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:43:20.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.893+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f0004eb0 con 0x7fd80c14be90 2024-03-25T19:43:20.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.900+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892842380 0 0) 0x7fd80c1d7840 con 0x7fd80c0b4d60 2024-03-25T19:43:20.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.900+0000 7fd802ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.900+0000 7fd802ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.900+0000 7fd802ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.900+0000 7fd802ffd640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:43:20.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.900+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd80c1d8470 con 0x7fd80c0b4d60 2024-03-25T19:43:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005051453 0 0) 0x7fd7f0004eb0 con 0x7fd80c14be90 2024-03-25T19:43:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd80c1d7840 con 0x7fd80c14be90 2024-03-25T19:43:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd804001b90 con 0x7fd80c14be90 2024-03-25T19:43:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:20.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.908+0000 7fd802ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4037842298 0 0) 0x7fd80c1d7840 con 0x7fd80c14be90 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 >> v1:172.21.15.142:6789/0 conn(0x7fd80c0b4d60 legacy=0x7fd80c1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 >> v1:172.21.15.47:6789/0 conn(0x7fd80c0b1730 legacy=0x7fd80c139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 1 monclient: found mon.c 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c1d8600 con 0x7fd80c14be90 2024-03-25T19:43:20.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd802ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 10 monclient: _renew_subs 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80c13afd0 con 0x7fd80c14be90 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 10 monclient: _renew_subs 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80c143340 con 0x7fd80c14be90 2024-03-25T19:43:20.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.917+0000 7fd81434b640 1 librados: init done 2024-03-25T19:43:20.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.918+0000 7fd81434b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:20.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.918+0000 7fd81434b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:20.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.918+0000 7fd81434b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:20.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:20.918+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80c0627b0 con 0x7fd80c14be90 2024-03-25T19:43:21.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd804002080 con 0x7fd80c14be90 2024-03-25T19:43:21.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd802ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:21.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fd804003b30 con 0x7fd80c14be90 2024-03-25T19:43:21.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd802ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd802ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd802ffd640 10 monclient: dump: 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:21.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.136+0000 7fd8108e4640 4 set_mon_vals no callback set 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.224+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fd804003c90 con 0x7fd80c14be90 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.225+0000 7fd8120e7640 10 monclient: get_auth_request con 0x7fd7f006dd10 auth_method 0 2024-03-25T19:43:21.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.225+0000 7fd802ffd640 10 client.4604.objecter ms_handle_connect 0x7fd7f006dd10 2024-03-25T19:43:21.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fd804053820 con 0x7fd80c14be90 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 10 client.4604.objecter ms_dispatch 0x7fd80c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd80404f160 con 0x7fd80c14be90 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:21.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.232+0000 7fd802ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:21.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.496+0000 7fd81434b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:21.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.496+0000 7fd81434b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:21.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.496+0000 7fd81434b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:21.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.496+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd80c1d79d0 con 0x7fd80c14be90 2024-03-25T19:43:21.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.519+0000 7fd802ffd640 1 -- 172.21.15.47:0/371231948 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fd804051260 con 0x7fd80c14be90 2024-03-25T19:43:21.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.519+0000 7fd802ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:21.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.519+0000 7fd802ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:21.519 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:21.519 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:21.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 10 librados: watch_flush enter 2024-03-25T19:43:21.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 10 librados: watch_flush exit 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 10 client.4604.objecter successfully canceled tick 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 >> v1:172.21.15.142:6800/1025302268 conn(0x7fd7f006dd10 legacy=0x7fd7f00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 10 monclient: shutdown 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 >> v1:172.21.15.47:6790/0 conn(0x7fd80c14be90 legacy=0x7fd80c1e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.521+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 shutdown_connections 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.522+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 >> 172.21.15.47:0/371231948 conn(0x7fd80c0617a0 msgr2=0x7fd80c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.522+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 shutdown_connections 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.522+0000 7fd81434b640 1 -- 172.21.15.47:0/371231948 wait complete. 2024-03-25T19:43:21.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:21.522+0000 7fd81434b640 1 librados: shutdown 2024-03-25T19:43:21.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:23.713 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:00.948 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.762 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:00.997 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.762 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:00.952 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.765 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:00.955 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.807 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:01.043 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.817 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:01.007 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.831 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:01.067 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.836 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:01.072 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:23.878 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:01.114 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:28.338 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:28.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.456+0000 7f1af6754640 10 monclient: get_monmap_and_config 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.456+0000 7f1af6754640 10 monclient: build_initial_monmap 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.456+0000 7f1af6754640 10 monclient: monmap: 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:28.457140+0000 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:28.457140+0000 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:28.456 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.457+0000 7f1af6754640 1 Processor -- start 2024-03-25T19:43:28.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.457+0000 7f1af6754640 1 -- start start 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.457+0000 7f1af6754640 10 monclient: init 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.457+0000 7f1af6754640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.457+0000 7f1af6754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 10 monclient(hunting): picked mon.noname-b con 0x7f1af00a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 10 monclient(hunting): picked mon.noname-c con 0x7f1af013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 10 monclient(hunting): picked mon.noname-a con 0x7f1af01430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0142520 con 0x7f1af01430c0 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00b28a0 con 0x7f1af013f710 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00b2a30 con 0x7f1af00a98f0 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:28.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.458+0000 7f1af6754640 10 monclient(hunting): authenticate will time out at 6123.744088s 2024-03-25T19:43:28.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.461+0000 7f1aef7fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1af013f710 0x7f1af01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:36734/0 (socket says 172.21.15.47:36734) 2024-03-25T19:43:28.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.461+0000 7f1aef7fe640 1 -- 172.21.15.47:0/2132711921 learned_addr learned my addr 172.21.15.47:0/2132711921 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:28.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.548+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234568839 0 0) 0x7f1af0142520 con 0x7f1af01430c0 2024-03-25T19:43:28.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.548+0000 7f1aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.548+0000 7f1aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.548+0000 7f1aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.548+0000 7f1aeeffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:43:28.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.548+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad8003520 con 0x7f1af01430c0 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2148210621 0 0) 0x7f1ad8003520 con 0x7f1af01430c0 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 >> v1:172.21.15.47:6790/0 conn(0x7f1af00a98f0 legacy=0x7f1af00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 >> v1:172.21.15.142:6789/0 conn(0x7f1af013f710 legacy=0x7f1af01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 1 monclient: found mon.noname-a 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00b44c0 con 0x7f1af01430c0 2024-03-25T19:43:28.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1ae8001a10 con 0x7f1af01430c0 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1af6754640 5 monclient: authenticate success, global_id 4617 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1af6754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1aeeffd640 10 monclient: dump: 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:28.575 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:28.576 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.574+0000 7f1af6754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.600+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae8001b70 con 0x7f1af01430c0 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.600+0000 7f1aeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.600+0000 7f1aeeffd640 1 -- 172.21.15.47:0/2132711921 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1ae8001f80 con 0x7f1af01430c0 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.600+0000 7f1af6754640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.600+0000 7f1af6754640 4 set_mon_vals no callback set 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.600+0000 7f1aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1aeeffd640 10 monclient: dump: 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:28.601 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1af6754640 10 monclient: shutdown 2024-03-25T19:43:28.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1af6754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:28.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1af6754640 1 -- 172.21.15.47:0/2132711921 >> v1:172.21.15.47:6789/0 conn(0x7f1af01430c0 legacy=0x7f1af014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1af6754640 1 -- 172.21.15.47:0/2132711921 shutdown_connections 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1af6754640 1 -- 172.21.15.47:0/2132711921 >> 172.21.15.47:0/2132711921 conn(0x7f1af00b7ea0 msgr2=0x7f1af00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.601+0000 7f1af6754640 1 -- 172.21.15.47:0/2132711921 shutdown_connections 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.602+0000 7f1af6754640 1 -- 172.21.15.47:0/2132711921 wait complete. 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.602+0000 7f1af6754640 10 monclient: build_initial_monmap 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.602+0000 7f1af6754640 10 monclient: monmap: 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 librados: starting msgr at 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 librados: starting objecter 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 Processor -- start 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 -- start start 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 librados: setting wanted keys 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 librados: calling monclient init 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 10 monclient: init 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 10 monclient(hunting): picked mon.noname-c con 0x7f1af00a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 10 monclient(hunting): picked mon.noname-a con 0x7f1af013f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 10 monclient(hunting): picked mon.noname-b con 0x7f1af01430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0138b60 con 0x7f1af013f710 2024-03-25T19:43:28.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.603+0000 7f1af6754640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0137e40 con 0x7f1af01430c0 2024-03-25T19:43:28.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.604+0000 7f1af6754640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af01d82f0 con 0x7f1af00a98f0 2024-03-25T19:43:28.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.604+0000 7f1af6754640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:28.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.604+0000 7f1af6754640 10 monclient(hunting): authenticate will time out at 6123.889956s 2024-03-25T19:43:28.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.617+0000 7f1af4cf1640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1af01430c0 0x7f1af0133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:36778/0 (socket says 172.21.15.47:36778) 2024-03-25T19:43:28.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.617+0000 7f1af4cf1640 1 -- 172.21.15.47:0/323340319 learned_addr learned my addr 172.21.15.47:0/323340319 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:28.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.620+0000 7f1aecff9640 10 client.?.objecter ms_handle_connect 0x7f1af01430c0 2024-03-25T19:43:28.619 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.620+0000 7f1aecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:28.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.624+0000 7f1aecff9640 10 client.?.objecter ms_handle_connect 0x7f1af00a98f0 2024-03-25T19:43:28.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.624+0000 7f1aecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:28.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.629+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893612840 0 0) 0x7f1af0137e40 con 0x7f1af01430c0 2024-03-25T19:43:28.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.629+0000 7f1aecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.629+0000 7f1aecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.629+0000 7f1aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.629+0000 7f1aecff9640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:43:28.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.629+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1adc004eb0 con 0x7f1af01430c0 2024-03-25T19:43:28.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.632+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154061200 0 0) 0x7f1af01d82f0 con 0x7f1af00a98f0 2024-03-25T19:43:28.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.632+0000 7f1aecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:28.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.632+0000 7f1aecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:28.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.632+0000 7f1aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:28.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.632+0000 7f1aecff9640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:43:28.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.632+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0137e40 con 0x7f1af00a98f0 2024-03-25T19:43:28.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.635+0000 7f1aecff9640 10 client.?.objecter ms_handle_connect 0x7f1af013f710 2024-03-25T19:43:28.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.635+0000 7f1aecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:28.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4104042101 0 0) 0x7f1af0137e40 con 0x7f1af00a98f0 2024-03-25T19:43:28.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af01d82f0 con 0x7f1af00a98f0 2024-03-25T19:43:28.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1ae00018a0 con 0x7f1af00a98f0 2024-03-25T19:43:28.643 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 10 monclient(hunting): dump: 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.644+0000 7f1aecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:28.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.653+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2921524038 0 0) 0x7f1adc004eb0 con 0x7f1af01430c0 2024-03-25T19:43:28.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.653+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af0137e40 con 0x7f1af01430c0 2024-03-25T19:43:28.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.653+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1ae8001fc0 con 0x7f1af01430c0 2024-03-25T19:43:28.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.653+0000 7f1aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.653+0000 7f1aecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.653+0000 7f1aecff9640 10 monclient(hunting): dump: 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:28.653 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.659+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018424957 0 0) 0x7f1af01d82f0 con 0x7f1af00a98f0 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 >> v1:172.21.15.142:6789/0 conn(0x7f1af01430c0 legacy=0x7f1af0133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 >> v1:172.21.15.47:6789/0 conn(0x7f1af013f710 legacy=0x7f1af01d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 1 monclient: found mon.c 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0134e70 con 0x7f1af00a98f0 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1aecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 5 monclient: authenticate success, global_id 4610 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 10 monclient: _renew_subs 2024-03-25T19:43:28.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af01350c0 con 0x7f1af00a98f0 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 10 monclient: _renew_subs 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af0134800 con 0x7f1af00a98f0 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.660+0000 7f1af6754640 1 librados: init done 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.661+0000 7f1af6754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.661+0000 7f1af6754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:28.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.661+0000 7f1af6754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:28.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.661+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af014f330 con 0x7f1af00a98f0 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae00035f0 con 0x7f1af00a98f0 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f1ae0003820 con 0x7f1af00a98f0 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aecff9640 10 monclient: dump: 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:28.957 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.957+0000 7f1aee7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.958+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f1ae0003a40 con 0x7f1af00a98f0 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.959+0000 7f1aef7fe640 10 monclient: get_auth_request con 0x7f1adc06df20 auth_method 0 2024-03-25T19:43:28.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.959+0000 7f1aecff9640 10 client.4610.objecter ms_handle_connect 0x7f1adc06df20 2024-03-25T19:43:28.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f1ae0053400 con 0x7f1af00a98f0 2024-03-25T19:43:28.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 10 client.4610.objecter ms_dispatch 0x7f1af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:28.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:28.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:28.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T19:43:28.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ae004ef10 con 0x7f1af00a98f0 2024-03-25T19:43:28.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:28.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:28.965+0000 7f1aecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:29.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.208+0000 7f1af6754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:29.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.208+0000 7f1af6754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:29.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.208+0000 7f1af6754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:29.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.208+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1af0074cc0 con 0x7f1af00a98f0 2024-03-25T19:43:29.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.240+0000 7f1aecff9640 1 -- 172.21.15.47:0/323340319 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f1ae0050e40 con 0x7f1af00a98f0 2024-03-25T19:43:29.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.240+0000 7f1aecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:29.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.240+0000 7f1aecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:29.240 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:29.240 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:29.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 10 librados: watch_flush enter 2024-03-25T19:43:29.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 10 librados: watch_flush exit 2024-03-25T19:43:29.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 10 client.4610.objecter successfully canceled tick 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 >> v1:172.21.15.142:6800/1025302268 conn(0x7f1adc06df20 legacy=0x7f1adc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 10 monclient: shutdown 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 >> v1:172.21.15.47:6790/0 conn(0x7f1af00a98f0 legacy=0x7f1af0137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 shutdown_connections 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 >> 172.21.15.47:0/323340319 conn(0x7f1af00b7ea0 msgr2=0x7f1af0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 shutdown_connections 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.242+0000 7f1af6754640 1 -- 172.21.15.47:0/323340319 wait complete. 2024-03-25T19:43:29.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:29.243+0000 7f1af6754640 1 librados: shutdown 2024-03-25T19:43:29.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:36.061 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:36.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.174+0000 7f560400f640 10 monclient: get_monmap_and_config 2024-03-25T19:43:36.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.174+0000 7f560400f640 10 monclient: build_initial_monmap 2024-03-25T19:43:36.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.174+0000 7f560400f640 10 monclient: monmap: 2024-03-25T19:43:36.174 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:36.174 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:36.175984+0000 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:36.175984+0000 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:36.175 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.174+0000 7f560400f640 1 Processor -- start 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 1 -- start start 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient: init 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient(hunting): picked mon.noname-a con 0x7f55fc0705b0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient(hunting): picked mon.noname-c con 0x7f55fc1402d0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:36.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient(hunting): picked mon.noname-b con 0x7f55fc149ae0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:36.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0718c0 con 0x7f55fc0705b0 2024-03-25T19:43:36.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0714c0 con 0x7f55fc1402d0 2024-03-25T19:43:36.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc072b50 con 0x7f55fc149ae0 2024-03-25T19:43:36.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:36.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.175+0000 7f560400f640 10 monclient(hunting): authenticate will time out at 6131.462770s 2024-03-25T19:43:36.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.184+0000 7f56015aa640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f55fc1402d0 0x7f55fc140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:41548/0 (socket says 172.21.15.47:41548) 2024-03-25T19:43:36.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.184+0000 7f56015aa640 1 -- 172.21.15.47:0/1727636558 learned_addr learned my addr 172.21.15.47:0/1727636558 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:36.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.193+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455870005 0 0) 0x7f55fc0714c0 con 0x7f55fc1402d0 2024-03-25T19:43:36.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.193+0000 7f5600da9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.193+0000 7f5600da9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.193+0000 7f5600da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.193+0000 7f5600da9640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:43:36.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.193+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e8003520 con 0x7f55fc1402d0 2024-03-25T19:43:36.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.202+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2659866318 0 0) 0x7f55e8003520 con 0x7f55fc1402d0 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.202+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 >> v1:172.21.15.47:6790/0 conn(0x7f55fc149ae0 legacy=0x7f55fc14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.202+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 >> v1:172.21.15.47:6789/0 conn(0x7f55fc0705b0 legacy=0x7f55fc13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 1 monclient: found mon.noname-c 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc13dd10 con 0x7f55fc1402d0 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: _finish_auth 0 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f55ec001690 con 0x7f55fc1402d0 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f560400f640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f560400f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f5600da9640 10 monclient: dump: 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:36.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.203+0000 7f560400f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:36.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55ec0017f0 con 0x7f55fc1402d0 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f5600da9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f5600da9640 1 -- 172.21.15.47:0/1727636558 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f55ec001c00 con 0x7f55fc1402d0 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f560400f640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f560400f640 4 set_mon_vals no callback set 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f5600da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f5600da9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.214+0000 7f5600da9640 10 monclient: dump: 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:36.216 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:36.217 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:36.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.215+0000 7f560400f640 10 monclient: shutdown 2024-03-25T19:43:36.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.215+0000 7f560400f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:36.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.215+0000 7f560400f640 1 -- 172.21.15.47:0/1727636558 >> v1:172.21.15.142:6789/0 conn(0x7f55fc1402d0 legacy=0x7f55fc140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.215+0000 7f560400f640 1 -- 172.21.15.47:0/1727636558 shutdown_connections 2024-03-25T19:43:36.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.215+0000 7f560400f640 1 -- 172.21.15.47:0/1727636558 >> 172.21.15.47:0/1727636558 conn(0x7f55fc0b5d10 msgr2=0x7f55fc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.216+0000 7f560400f640 1 -- 172.21.15.47:0/1727636558 shutdown_connections 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.216+0000 7f560400f640 1 -- 172.21.15.47:0/1727636558 wait complete. 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 10 monclient: build_initial_monmap 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 10 monclient: monmap: 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:36.218 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 1 librados: starting msgr at 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 1 librados: starting objecter 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 1 Processor -- start 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 1 -- start start 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 1 librados: setting wanted keys 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 1 librados: calling monclient init 2024-03-25T19:43:36.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.217+0000 7f560400f640 10 monclient: init 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient(hunting): picked mon.noname-b con 0x7f55fc0705b0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient(hunting): picked mon.noname-a con 0x7f55fc1402d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient(hunting): picked mon.noname-c con 0x7f55fc149ae0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc1467f0 con 0x7f55fc1402d0 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc146ad0 con 0x7f55fc0705b0 2024-03-25T19:43:36.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc145690 con 0x7f55fc149ae0 2024-03-25T19:43:36.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:36.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.218+0000 7f560400f640 10 monclient(hunting): authenticate will time out at 6131.505817s 2024-03-25T19:43:36.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.224+0000 7f5601dab640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f55fc0705b0 0x7f55fc144ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:41568/0 (socket says 172.21.15.47:41568) 2024-03-25T19:43:36.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.224+0000 7f5601dab640 1 -- 172.21.15.47:0/3194285272 learned_addr learned my addr 172.21.15.47:0/3194285272 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:36.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.224+0000 7f55f27fc640 10 client.?.objecter ms_handle_connect 0x7f55fc0705b0 2024-03-25T19:43:36.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.224+0000 7f55f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:36.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.227+0000 7f55f27fc640 10 client.?.objecter ms_handle_connect 0x7f55fc149ae0 2024-03-25T19:43:36.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.227+0000 7f55f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:36.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.229+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328932038 0 0) 0x7f55fc146ad0 con 0x7f55fc0705b0 2024-03-25T19:43:36.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.229+0000 7f55f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.229+0000 7f55f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.229+0000 7f55f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.229+0000 7f55f27fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:43:36.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.229+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e0004eb0 con 0x7f55fc0705b0 2024-03-25T19:43:36.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 10 client.?.objecter ms_handle_connect 0x7f55fc1402d0 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3822157081 0 0) 0x7f55e0004eb0 con 0x7f55fc0705b0 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55fc146ad0 con 0x7f55fc0705b0 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f55f8001a30 con 0x7f55fc0705b0 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 10 monclient(hunting): dump: 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:36.236 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:36.237 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:36.237 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:36.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.235+0000 7f55f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:36.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.237+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429419354 0 0) 0x7f55fc145690 con 0x7f55fc149ae0 2024-03-25T19:43:36.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.237+0000 7f55f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.237+0000 7f55f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.237+0000 7f55f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.237+0000 7f55f27fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:43:36.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.237+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e0004eb0 con 0x7f55fc149ae0 2024-03-25T19:43:36.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.242+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030179955 0 0) 0x7f55fc1467f0 con 0x7f55fc1402d0 2024-03-25T19:43:36.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.242+0000 7f55f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:36.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.242+0000 7f55f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:36.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.242+0000 7f55f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:36.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.242+0000 7f55f27fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:43:36.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.242+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc145690 con 0x7f55fc1402d0 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 759460569 0 0) 0x7f55fc146ad0 con 0x7f55fc0705b0 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 >> v1:172.21.15.47:6790/0 conn(0x7f55fc149ae0 legacy=0x7f55fc148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 >> v1:172.21.15.47:6789/0 conn(0x7f55fc1402d0 legacy=0x7f55fc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 1 monclient: found mon.b 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc1443b0 con 0x7f55fc0705b0 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f55f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 10 monclient: _renew_subs 2024-03-25T19:43:36.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc144600 con 0x7f55fc0705b0 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 10 monclient: _renew_subs 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc142310 con 0x7f55fc0705b0 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.245+0000 7f560400f640 1 librados: init done 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.246+0000 7f560400f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.246+0000 7f560400f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:36.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.246+0000 7f560400f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:36.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.246+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc145870 con 0x7f55fc0705b0 2024-03-25T19:43:36.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55f8001d70 con 0x7f55fc0705b0 2024-03-25T19:43:36.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:36.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f55f80038c0 con 0x7f55fc0705b0 2024-03-25T19:43:36.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 10 monclient: dump: 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:36.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f3fff640 4 set_mon_vals no callback set 2024-03-25T19:43:36.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.518+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f55f8003ae0 con 0x7f55fc0705b0 2024-03-25T19:43:36.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.519+0000 7f56015aa640 10 monclient: get_auth_request con 0x7f55e006dea0 auth_method 0 2024-03-25T19:43:36.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.519+0000 7f55f27fc640 10 client.4627.objecter ms_handle_connect 0x7f55e006dea0 2024-03-25T19:43:36.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.523+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f55f80546c0 con 0x7f55fc0705b0 2024-03-25T19:43:36.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.523+0000 7f55f27fc640 10 client.4627.objecter ms_dispatch 0x7f55fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:36.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.523+0000 7f55f27fc640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:36.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.523+0000 7f55f27fc640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:36.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.523+0000 7f55f27fc640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T19:43:36.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.525+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55f80501d0 con 0x7f55fc0705b0 2024-03-25T19:43:36.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.526+0000 7f55f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:36.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.526+0000 7f55f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:36.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.745+0000 7f560400f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:36.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.745+0000 7f560400f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:36.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.745+0000 7f560400f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:36.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.745+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55fc0627b0 con 0x7f55fc0705b0 2024-03-25T19:43:36.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.818+0000 7f55f27fc640 1 -- 172.21.15.47:0/3194285272 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f55f8052100 con 0x7f55fc0705b0 2024-03-25T19:43:36.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.818+0000 7f55f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:36.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.818+0000 7f55f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:36.818 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:36.818 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 10 librados: watch_flush enter 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 10 librados: watch_flush exit 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 10 client.4627.objecter successfully canceled tick 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 >> v1:172.21.15.142:6800/1025302268 conn(0x7f55e006dea0 legacy=0x7f55e0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 10 monclient: shutdown 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:36.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 >> v1:172.21.15.142:6789/0 conn(0x7f55fc0705b0 legacy=0x7f55fc144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:36.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 shutdown_connections 2024-03-25T19:43:36.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 >> 172.21.15.47:0/3194285272 conn(0x7f55fc0b5d10 msgr2=0x7f55fc140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:36.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 shutdown_connections 2024-03-25T19:43:36.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 -- 172.21.15.47:0/3194285272 wait complete. 2024-03-25T19:43:36.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:36.820+0000 7f560400f640 1 librados: shutdown 2024-03-25T19:43:36.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:43.638 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.748+0000 7f4b04fc8640 10 monclient: get_monmap_and_config 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.748+0000 7f4b04fc8640 10 monclient: build_initial_monmap 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.748+0000 7f4b04fc8640 10 monclient: monmap: 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:43.749788+0000 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:43.749788+0000 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:43.748 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.748+0000 7f4b04fc8640 1 Processor -- start 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.749+0000 7f4b04fc8640 1 -- start start 2024-03-25T19:43:43.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.749+0000 7f4b04fc8640 10 monclient: init 2024-03-25T19:43:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.749+0000 7f4b04fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.749+0000 7f4b04fc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f4b000b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f4b000b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0014be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000af940 con 0x7f4b000b1730 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000b0a70 con 0x7f4b000b4d60 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000b0d50 con 0x7f4b0014be90 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4b04fc8640 10 monclient(hunting): authenticate will time out at 6139.037119s 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4afed9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4b0014be90 0x7f4b0014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48044/0 (socket says 172.21.15.47:48044) 2024-03-25T19:43:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.750+0000 7f4afed9d640 1 -- 172.21.15.47:0/758909914 learned_addr learned my addr 172.21.15.47:0/758909914 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:43.751 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:20.986 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.805 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:21.041 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.827 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:21.063 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.852 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:21.042 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.859+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169246174 0 0) 0x7f4b000b0a70 con 0x7f4b000b4d60 2024-03-25T19:43:43.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.860+0000 7f4afd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:43.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.860+0000 7f4afd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:43.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.860+0000 7f4afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:43.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.860+0000 7f4afd59a640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:43:43.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.860+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae8003520 con 0x7f4b000b4d60 2024-03-25T19:43:43.862 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:21.051 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.866+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608456282 0 0) 0x7f4b000b0d50 con 0x7f4b0014be90 2024-03-25T19:43:43.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.866+0000 7f4afd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:43.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.866+0000 7f4afd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:43.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.866+0000 7f4afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:43.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.866+0000 7f4afd59a640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:43:43.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.866+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b000b0a70 con 0x7f4b0014be90 2024-03-25T19:43:43.868 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:21.104 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.878 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:21.114 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217615 0 0) 0x7f4b000b0a70 con 0x7f4b0014be90 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 >> v1:172.21.15.142:6789/0 conn(0x7f4b000b4d60 legacy=0x7f4b0013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 >> v1:172.21.15.47:6789/0 conn(0x7f4b000b1730 legacy=0x7f4b000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 1 monclient: found mon.noname-b 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:43.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b000b1030 con 0x7f4b0014be90 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: _finish_auth 0 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4b04fc8640 5 monclient: authenticate success, global_id 4619 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4af80019f0 con 0x7f4b0014be90 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4b04fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4afd59a640 10 monclient: dump: 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:43.892 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:43.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.890+0000 7f4b04fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:43.899 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:21.089 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.917 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:21.153 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:43.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4af8001b50 con 0x7f4b0014be90 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4afd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4afd59a640 1 -- 172.21.15.47:0/758909914 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4af8001f20 con 0x7f4b0014be90 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4b04fc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4b04fc8640 4 set_mon_vals no callback set 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4afd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4afd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.944+0000 7f4afd59a640 10 monclient: dump: 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:43.945 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:43.946 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:43.946 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:43.946 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:43.946 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:43.946 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 10 monclient: shutdown 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 1 -- 172.21.15.47:0/758909914 >> v1:172.21.15.47:6790/0 conn(0x7f4b0014be90 legacy=0x7f4b0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 1 -- 172.21.15.47:0/758909914 shutdown_connections 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 1 -- 172.21.15.47:0/758909914 >> 172.21.15.47:0/758909914 conn(0x7f4b000617a0 msgr2=0x7f4b00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 1 -- 172.21.15.47:0/758909914 shutdown_connections 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.945+0000 7f4b04fc8640 1 -- 172.21.15.47:0/758909914 wait complete. 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 10 monclient: build_initial_monmap 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 10 monclient: monmap: 2024-03-25T19:43:43.948 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 1 librados: starting msgr at 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 1 librados: starting objecter 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 1 Processor -- start 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 1 -- start start 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 1 librados: setting wanted keys 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 1 librados: calling monclient init 2024-03-25T19:43:43.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.946+0000 7f4b04fc8640 10 monclient: init 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f4b000b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f4b000b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0014be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b00140920 con 0x7f4b0014be90 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b00137e90 con 0x7f4b000b1730 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b00138020 con 0x7f4b000b4d60 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:43.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.947+0000 7f4b04fc8640 10 monclient(hunting): authenticate will time out at 6139.234437s 2024-03-25T19:43:43.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.970+0000 7f4afed9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4b0014be90 0x7f4b001e2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33210/0 (socket says 172.21.15.47:33210) 2024-03-25T19:43:43.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.970+0000 7f4afed9d640 1 -- 172.21.15.47:0/3331483780 learned_addr learned my addr 172.21.15.47:0/3331483780 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:43.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.974+0000 7f4aeeffd640 10 client.?.objecter ms_handle_connect 0x7f4b0014be90 2024-03-25T19:43:43.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:43.974+0000 7f4aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.008+0000 7f4aeeffd640 10 client.?.objecter ms_handle_connect 0x7f4b000b1730 2024-03-25T19:43:44.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.008+0000 7f4aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.012+0000 7f4aeeffd640 10 client.?.objecter ms_handle_connect 0x7f4b000b4d60 2024-03-25T19:43:44.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.012+0000 7f4aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.021+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993788536 0 0) 0x7f4b00140920 con 0x7f4b0014be90 2024-03-25T19:43:44.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.021+0000 7f4aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.021+0000 7f4aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.021+0000 7f4aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.021+0000 7f4aeeffd640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:43:44.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.021+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae0004eb0 con 0x7f4b0014be90 2024-03-25T19:43:44.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.026+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353343940 0 0) 0x7f4b00137e90 con 0x7f4b000b1730 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.026+0000 7f4aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.026+0000 7f4aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.026+0000 7f4aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.026+0000 7f4aeeffd640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:43:44.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.026+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b00140920 con 0x7f4b000b1730 2024-03-25T19:43:44.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.037+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232497097 0 0) 0x7f4b00138020 con 0x7f4b000b4d60 2024-03-25T19:43:44.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.037+0000 7f4aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.037+0000 7f4aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.037+0000 7f4aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.037+0000 7f4aeeffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:43:44.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.037+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b00137e90 con 0x7f4b000b4d60 2024-03-25T19:43:44.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3533267617 0 0) 0x7f4b00140920 con 0x7f4b000b1730 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b00138020 con 0x7f4b000b1730 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4af0001950 con 0x7f4b000b1730 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 10 monclient(hunting): dump: 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:44.043 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:44.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:44.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4afdd9b640 1 -- 172.21.15.47:0/3331483780 >> v1:172.21.15.47:6790/0 conn(0x7f4b000b4d60 legacy=0x7f4b001dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:43:44.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4afdd9b640 1 -- 172.21.15.47:0/3331483780 >> v1:172.21.15.47:6790/0 conn(0x7f4b000b4d60 legacy=0x7f4b001dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:43:44.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4afdd9b640 1 --1- 172.21.15.47:0/3331483780 >> v1:172.21.15.47:6790/0 conn(0x7f4b000b4d60 0x7f4b001dab20 :-1 s=OPENED pgs=66834 cs=1 l=1).handle_message read tag failed 2024-03-25T19:43:44.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4afdd9b640 1 --1- 172.21.15.47:0/3331483780 >> v1:172.21.15.47:6790/0 conn(0x7f4b000b4d60 0x7f4b001dab20 :-1 s=OPENED pgs=66834 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:43:44.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.042+0000 7f4aeeffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-03-25T19:43:44.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882544315 0 0) 0x7f4b00138020 con 0x7f4b000b1730 2024-03-25T19:43:44.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 >> v1:172.21.15.47:6790/0 conn(0x7f4b000b4d60 legacy=0x7f4b001dab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:43:44.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 >> v1:172.21.15.47:6789/0 conn(0x7f4b0014be90 legacy=0x7f4b001e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:44.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 1 monclient: found mon.b 2024-03-25T19:43:44.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b00138270 con 0x7f4b000b1730 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 10 monclient: _renew_subs 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b00138700 con 0x7f4b000b1730 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 10 monclient: _renew_subs 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b00132790 con 0x7f4b000b1730 2024-03-25T19:43:44.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.061+0000 7f4b04fc8640 1 librados: init done 2024-03-25T19:43:44.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.062+0000 7f4b04fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.062+0000 7f4b04fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.062+0000 7f4b04fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:44.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.062+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b000627b0 con 0x7f4b000b1730 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4af0003610 con 0x7f4b000b1730 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4aeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4af0003920 con 0x7f4b000b1730 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4aeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4aeeffd640 10 monclient: dump: 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:44.338 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:44.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.337+0000 7f4afcd99640 4 set_mon_vals no callback set 2024-03-25T19:43:44.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.338+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f4af0003b40 con 0x7f4b000b1730 2024-03-25T19:43:44.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.339+0000 7f4afdd9b640 10 monclient: get_auth_request con 0x7f4ae006de00 auth_method 0 2024-03-25T19:43:44.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.339+0000 7f4aeeffd640 10 client.4633.objecter ms_handle_connect 0x7f4ae006de00 2024-03-25T19:43:44.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.344+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f4af0053620 con 0x7f4b000b1730 2024-03-25T19:43:44.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.344+0000 7f4aeeffd640 10 client.4633.objecter ms_dispatch 0x7f4b00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:44.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.344+0000 7f4aeeffd640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:44.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.344+0000 7f4aeeffd640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:44.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.344+0000 7f4aeeffd640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T19:43:44.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.345+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4af004f130 con 0x7f4b000b1730 2024-03-25T19:43:44.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.345+0000 7f4aeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.345+0000 7f4aeeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:44.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.606+0000 7f4b04fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:44.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.606+0000 7f4b04fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:44.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.606+0000 7f4b04fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:43:44.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.606+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b00069f40 con 0x7f4b000b1730 2024-03-25T19:43:44.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.631+0000 7f4aeeffd640 1 -- 172.21.15.47:0/3331483780 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f4af0051060 con 0x7f4b000b1730 2024-03-25T19:43:44.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.631+0000 7f4aeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:44.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.631+0000 7f4aeeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:44.632 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:44.632 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 10 librados: watch_flush enter 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 10 librados: watch_flush exit 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 10 client.4633.objecter successfully canceled tick 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 >> v1:172.21.15.142:6800/1025302268 conn(0x7f4ae006de00 legacy=0x7f4ae00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 10 monclient: shutdown 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 >> v1:172.21.15.142:6789/0 conn(0x7f4b000b1730 legacy=0x7f4b00131ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.635 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 shutdown_connections 2024-03-25T19:43:44.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 >> 172.21.15.47:0/3331483780 conn(0x7f4b000617a0 msgr2=0x7f4b00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:44.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 shutdown_connections 2024-03-25T19:43:44.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 -- 172.21.15.47:0/3331483780 wait complete. 2024-03-25T19:43:44.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:44.634+0000 7f4b04fc8640 1 librados: shutdown 2024-03-25T19:43:44.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:51.454 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:51.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.575+0000 7f289c896640 10 monclient: get_monmap_and_config 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.575+0000 7f289c896640 10 monclient: build_initial_monmap 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.575+0000 7f289c896640 10 monclient: monmap: 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:51.576058+0000 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:51.576058+0000 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:51.575 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:51.576 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.576+0000 7f289c896640 1 Processor -- start 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.576+0000 7f289c896640 1 -- start start 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.576+0000 7f289c896640 10 monclient: init 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient(hunting): picked mon.noname-c con 0x7f28980a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient(hunting): picked mon.noname-b con 0x7f289813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient(hunting): picked mon.noname-a con 0x7f28981430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2898142520 con 0x7f28981430c0 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28980b28a0 con 0x7f28980a98f0 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28980b2a30 con 0x7f289813f710 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:51.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.577+0000 7f289c896640 10 monclient(hunting): authenticate will time out at 6146.863127s 2024-03-25T19:43:51.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.587+0000 7f2896d9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f28981430c0 0x7f289814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60052/0 (socket says 172.21.15.47:60052) 2024-03-25T19:43:51.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.587+0000 7f289659c640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f28980a98f0 0x7f28980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:41232/0 (socket says 172.21.15.47:41232) 2024-03-25T19:43:51.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.587+0000 7f2896d9d640 1 -- 172.21.15.47:0/566109163 learned_addr learned my addr 172.21.15.47:0/566109163 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:51.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.593+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773090871 0 0) 0x7f28980b2a30 con 0x7f289813f710 2024-03-25T19:43:51.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.593+0000 7f2895d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.593+0000 7f2895d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.593+0000 7f2895d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.593+0000 7f2895d9b640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:43:51.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.593+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2884003520 con 0x7f289813f710 2024-03-25T19:43:51.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.600+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2134418823 0 0) 0x7f2884003520 con 0x7f289813f710 2024-03-25T19:43:51.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 >> v1:172.21.15.142:6789/0 conn(0x7f28980a98f0 legacy=0x7f28980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 >> v1:172.21.15.47:6789/0 conn(0x7f28981430c0 legacy=0x7f289814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 1 monclient: found mon.noname-b 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28980b44c0 con 0x7f289813f710 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2880001690 con 0x7f289813f710 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f289c896640 5 monclient: authenticate success, global_id 4625 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f289c896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f2895d9b640 10 monclient: dump: 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:51.601 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:51.602 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.601+0000 7f289c896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:51.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.605+0000 7f2895d9b640 1 -- 172.21.15.47:0/566109163 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28800017f0 con 0x7f289813f710 2024-03-25T19:43:51.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.605+0000 7f2895d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:51.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.605+0000 7f289c896640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:51.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.605+0000 7f289c896640 4 set_mon_vals no callback set 2024-03-25T19:43:51.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 10 monclient: shutdown 2024-03-25T19:43:51.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:51.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 1 -- 172.21.15.47:0/566109163 >> v1:172.21.15.47:6790/0 conn(0x7f289813f710 legacy=0x7f28981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 1 -- 172.21.15.47:0/566109163 shutdown_connections 2024-03-25T19:43:51.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 1 -- 172.21.15.47:0/566109163 >> 172.21.15.47:0/566109163 conn(0x7f28980b7ea0 msgr2=0x7f28980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:51.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 1 -- 172.21.15.47:0/566109163 shutdown_connections 2024-03-25T19:43:51.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.606+0000 7f289c896640 1 -- 172.21.15.47:0/566109163 wait complete. 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 10 monclient: build_initial_monmap 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 10 monclient: monmap: 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:51.607 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 1 librados: starting msgr at 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 1 librados: starting objecter 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 1 Processor -- start 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 1 -- start start 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 1 librados: setting wanted keys 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 1 librados: calling monclient init 2024-03-25T19:43:51.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.607+0000 7f289c896640 10 monclient: init 2024-03-25T19:43:51.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:51.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:51.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient(hunting): picked mon.noname-a con 0x7f28980a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:51.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient(hunting): picked mon.noname-c con 0x7f289813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:51.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient(hunting): picked mon.noname-b con 0x7f28981430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:51.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289812f460 con 0x7f28980a98f0 2024-03-25T19:43:51.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289812ff70 con 0x7f28981430c0 2024-03-25T19:43:51.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289813c3a0 con 0x7f289813f710 2024-03-25T19:43:51.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:51.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.608+0000 7f289c896640 10 monclient(hunting): authenticate will time out at 6146.894613s 2024-03-25T19:43:51.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.617+0000 7f289659c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f28980a98f0 0x7f2898139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60080/0 (socket says 172.21.15.47:60080) 2024-03-25T19:43:51.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.617+0000 7f289659c640 1 -- 172.21.15.47:0/3759321015 learned_addr learned my addr 172.21.15.47:0/3759321015 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:51.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.623+0000 7f288effd640 10 client.?.objecter ms_handle_connect 0x7f289813f710 2024-03-25T19:43:51.623 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.623+0000 7f288effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:51.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.626+0000 7f288effd640 10 client.?.objecter ms_handle_connect 0x7f28980a98f0 2024-03-25T19:43:51.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.626+0000 7f288effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.631+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373806359 0 0) 0x7f289813c3a0 con 0x7f289813f710 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.631+0000 7f288effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.631+0000 7f288effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.631+0000 7f288effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.631+0000 7f288effd640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.631+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878004eb0 con 0x7f289813f710 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.632+0000 7f288effd640 10 client.?.objecter ms_handle_connect 0x7f28981430c0 2024-03-25T19:43:51.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.632+0000 7f288effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:51.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.639+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28911923 0 0) 0x7f289812f460 con 0x7f28980a98f0 2024-03-25T19:43:51.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.639+0000 7f288effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.639+0000 7f288effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.639+0000 7f288effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.639+0000 7f288effd640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:43:51.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.639+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289813c3a0 con 0x7f28980a98f0 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.645+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 430895882 0 0) 0x7f2878004eb0 con 0x7f289813f710 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.645+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f289812f460 con 0x7f289813f710 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.645+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f28800015d0 con 0x7f289813f710 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): dump: 2024-03-25T19:43:51.645 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:51.646 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:51.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283785618 0 0) 0x7f289812ff70 con 0x7f28981430c0 2024-03-25T19:43:51.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:51.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:51.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:51.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:43:51.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.646+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878004eb0 con 0x7f28981430c0 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2958711207 0 0) 0x7f289812f460 con 0x7f289813f710 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 >> v1:172.21.15.142:6789/0 conn(0x7f28981430c0 legacy=0x7f2898130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 >> v1:172.21.15.47:6789/0 conn(0x7f28980a98f0 legacy=0x7f2898139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 1 monclient: found mon.c 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289812f6d0 con 0x7f289813f710 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 10 monclient: _finish_auth 0 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f288effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:51.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 10 monclient: _renew_subs 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289812f9b0 con 0x7f289813f710 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 10 monclient: _renew_subs 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28981311b0 con 0x7f289813f710 2024-03-25T19:43:51.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.655+0000 7f289c896640 1 librados: init done 2024-03-25T19:43:51.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.656+0000 7f289c896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:51.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.656+0000 7f289c896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:51.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.656+0000 7f289c896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:51.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.656+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28980627b0 con 0x7f289813f710 2024-03-25T19:43:51.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28800038a0 con 0x7f289813f710 2024-03-25T19:43:51.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f288effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:51.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f2880001250 con 0x7f289813f710 2024-03-25T19:43:51.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f288effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:51.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f288effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f288effd640 10 monclient: dump: 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:51.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.978+0000 7f289559a640 4 set_mon_vals no callback set 2024-03-25T19:43:51.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.979+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f2880019aa0 con 0x7f289813f710 2024-03-25T19:43:51.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.980+0000 7f289659c640 10 monclient: get_auth_request con 0x7f287806ddb0 auth_method 0 2024-03-25T19:43:51.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.980+0000 7f288effd640 10 client.4628.objecter ms_handle_connect 0x7f287806ddb0 2024-03-25T19:43:51.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.990+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f2880053760 con 0x7f289813f710 2024-03-25T19:43:51.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.990+0000 7f288effd640 10 client.4628.objecter ms_dispatch 0x7f2898000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:51.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.990+0000 7f288effd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:51.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.990+0000 7f288effd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:51.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.991+0000 7f288effd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T19:43:51.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.991+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f288004f0a0 con 0x7f289813f710 2024-03-25T19:43:51.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.991+0000 7f288effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:51.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:51.991+0000 7f288effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:52.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.252+0000 7f289c896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:52.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.252+0000 7f289c896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:52.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.253+0000 7f289c896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:52.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.253+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2898069f40 con 0x7f289813f710 2024-03-25T19:43:52.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.283+0000 7f288effd640 1 -- 172.21.15.47:0/3759321015 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f28800511a0 con 0x7f289813f710 2024-03-25T19:43:52.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.283+0000 7f288effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:52.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.283+0000 7f288effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:52.283 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:52.283 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 10 librados: watch_flush enter 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 10 librados: watch_flush exit 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 10 client.4628.objecter successfully canceled tick 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 >> v1:172.21.15.142:6800/1025302268 conn(0x7f287806ddb0 legacy=0x7f2878070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 10 monclient: shutdown 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:52.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 >> v1:172.21.15.47:6790/0 conn(0x7f289813f710 legacy=0x7f2898130390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 shutdown_connections 2024-03-25T19:43:52.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 >> 172.21.15.47:0/3759321015 conn(0x7f28980b7ea0 msgr2=0x7f2898074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:52.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 shutdown_connections 2024-03-25T19:43:52.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 -- 172.21.15.47:0/3759321015 wait complete. 2024-03-25T19:43:52.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:52.286+0000 7f289c896640 1 librados: shutdown 2024-03-25T19:43:52.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:59.105 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:59.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.220+0000 7fa1d9939640 10 monclient: get_monmap_and_config 2024-03-25T19:43:59.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.220+0000 7fa1d9939640 10 monclient: build_initial_monmap 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.220+0000 7fa1d9939640 10 monclient: monmap: 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:43:59.221166+0000 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:43:59.221166+0000 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:43:59.220 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 1 Processor -- start 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 1 -- start start 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient: init 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d40a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:59.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d41430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:59.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4142520 con 0x7fa1d41430c0 2024-03-25T19:43:59.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40b28a0 con 0x7fa1d413f710 2024-03-25T19:43:59.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40b2a30 con 0x7fa1d40a98f0 2024-03-25T19:43:59.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:59.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.221+0000 7fa1d9939640 10 monclient(hunting): authenticate will time out at 6154.507844s 2024-03-25T19:43:59.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.244+0000 7fa1d27fc640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa1d413f710 0x7fa1d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:54338/0 (socket says 172.21.15.47:54338) 2024-03-25T19:43:59.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.244+0000 7fa1d27fc640 1 -- 172.21.15.47:0/4034669509 learned_addr learned my addr 172.21.15.47:0/4034669509 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:59.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.254+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159987867 0 0) 0x7fa1d4142520 con 0x7fa1d41430c0 2024-03-25T19:43:59.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.254+0000 7fa1d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.254+0000 7fa1d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.254+0000 7fa1d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.254+0000 7fa1d1ffb640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:43:59.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.254+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc003520 con 0x7fa1d41430c0 2024-03-25T19:43:59.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.261+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316387595 0 0) 0x7fa1d40b2a30 con 0x7fa1d40a98f0 2024-03-25T19:43:59.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.261+0000 7fa1d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.261+0000 7fa1d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.261+0000 7fa1d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.261+0000 7fa1d1ffb640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:43:59.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.261+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4142520 con 0x7fa1d40a98f0 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113479959 0 0) 0x7fa1bc003520 con 0x7fa1d41430c0 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 >> v1:172.21.15.47:6790/0 conn(0x7fa1d40a98f0 legacy=0x7fa1d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 >> v1:172.21.15.142:6789/0 conn(0x7fa1d413f710 legacy=0x7fa1d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 1 monclient: found mon.noname-a 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d40b44c0 con 0x7fa1d41430c0 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:59.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa1cc001a10 con 0x7fa1d41430c0 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d1ffb640 10 monclient: dump: 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:59.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d9939640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:43:59.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.271+0000 7fa1d9939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:59.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1cc001b70 con 0x7fa1d41430c0 2024-03-25T19:43:59.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:59.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d1ffb640 1 -- 172.21.15.47:0/4034669509 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa1cc001f80 con 0x7fa1d41430c0 2024-03-25T19:43:59.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d1ffb640 10 monclient: dump: 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d9939640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:59.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.276+0000 7fa1d9939640 4 set_mon_vals no callback set 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 10 monclient: shutdown 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 1 -- 172.21.15.47:0/4034669509 >> v1:172.21.15.47:6789/0 conn(0x7fa1d41430c0 legacy=0x7fa1d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 1 -- 172.21.15.47:0/4034669509 shutdown_connections 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 1 -- 172.21.15.47:0/4034669509 >> 172.21.15.47:0/4034669509 conn(0x7fa1d40b7ea0 msgr2=0x7fa1d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 1 -- 172.21.15.47:0/4034669509 shutdown_connections 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.277+0000 7fa1d9939640 1 -- 172.21.15.47:0/4034669509 wait complete. 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 10 monclient: build_initial_monmap 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 10 monclient: monmap: 2024-03-25T19:43:59.278 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 1 librados: starting msgr at 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 1 librados: starting objecter 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 1 Processor -- start 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 1 -- start start 2024-03-25T19:43:59.279 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 1 librados: setting wanted keys 2024-03-25T19:43:59.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 1 librados: calling monclient init 2024-03-25T19:43:59.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.278+0000 7fa1d9939640 10 monclient: init 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.279+0000 7fa1d9939640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.279+0000 7fa1d9939640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.279+0000 7fa1d9939640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d40a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.279+0000 7fa1d9939640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d413f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.279+0000 7fa1d9939640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d41430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.280+0000 7fa1d9939640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d413b410 con 0x7fa1d41430c0 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.280+0000 7fa1d9939640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d413b010 con 0x7fa1d40a98f0 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.280+0000 7fa1d9939640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4135230 con 0x7fa1d413f710 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.280+0000 7fa1d9939640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:59.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.280+0000 7fa1d9939640 10 monclient(hunting): authenticate will time out at 6154.566131s 2024-03-25T19:43:59.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.285+0000 7fa1d27fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa1d413f710 0x7fa1d41dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47388/0 (socket says 172.21.15.47:47388) 2024-03-25T19:43:59.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.285+0000 7fa1d27fc640 1 -- 172.21.15.47:0/4266540998 learned_addr learned my addr 172.21.15.47:0/4266540998 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:43:59.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.287+0000 7fa1bbfff640 10 client.?.objecter ms_handle_connect 0x7fa1d413f710 2024-03-25T19:43:59.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.287+0000 7fa1bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:59.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.296+0000 7fa1bbfff640 10 client.?.objecter ms_handle_connect 0x7fa1d41430c0 2024-03-25T19:43:59.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.296+0000 7fa1bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:59.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.303+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182180204 0 0) 0x7fa1d4135230 con 0x7fa1d413f710 2024-03-25T19:43:59.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.303+0000 7fa1bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.303+0000 7fa1bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.303+0000 7fa1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.303+0000 7fa1bbfff640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:43:59.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.303+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c0004eb0 con 0x7fa1d413f710 2024-03-25T19:43:59.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.310+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942366310 0 0) 0x7fa1d413b410 con 0x7fa1d41430c0 2024-03-25T19:43:59.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.310+0000 7fa1bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.310+0000 7fa1bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.310+0000 7fa1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.310+0000 7fa1bbfff640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:43:59.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.310+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4135230 con 0x7fa1d41430c0 2024-03-25T19:43:59.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.321+0000 7fa1bbfff640 10 client.?.objecter ms_handle_connect 0x7fa1d40a98f0 2024-03-25T19:43:59.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.321+0000 7fa1bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136835014 0 0) 0x7fa1c0004eb0 con 0x7fa1d413f710 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d413b410 con 0x7fa1d413f710 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa1c80018b0 con 0x7fa1d413f710 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 10 monclient(hunting): dump: 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.327 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:59.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.327+0000 7fa1bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:59.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.348+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418939402 0 0) 0x7fa1d413b010 con 0x7fa1d40a98f0 2024-03-25T19:43:59.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.348+0000 7fa1bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:59.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.348+0000 7fa1bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:59.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.348+0000 7fa1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:59.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.348+0000 7fa1bbfff640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:43:59.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.348+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c0004eb0 con 0x7fa1d40a98f0 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1344002061 0 0) 0x7fa1d413b410 con 0x7fa1d413f710 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 >> v1:172.21.15.142:6789/0 conn(0x7fa1d40a98f0 legacy=0x7fa1d4130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 >> v1:172.21.15.47:6789/0 conn(0x7fa1d41430c0 legacy=0x7fa1d41db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 1 monclient: found mon.c 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d41353c0 con 0x7fa1d413f710 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:59.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 5 monclient: authenticate success, global_id 4634 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 10 monclient: _renew_subs 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d40b78e0 con 0x7fa1d413f710 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 10 monclient: _renew_subs 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d41317d0 con 0x7fa1d413f710 2024-03-25T19:43:59.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.350+0000 7fa1d9939640 1 librados: init done 2024-03-25T19:43:59.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.351+0000 7fa1d9939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:59.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.351+0000 7fa1d9939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:59.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.351+0000 7fa1d9939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:59.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.351+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d40b7c80 con 0x7fa1d413f710 2024-03-25T19:43:59.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c80047e0 con 0x7fa1d413f710 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa1c8004c30 con 0x7fa1d413f710 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1bbfff640 10 monclient: dump: 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:43:59.660 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.660+0000 7fa1d17fa640 4 set_mon_vals no callback set 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.661+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fa1c8004e50 con 0x7fa1d413f710 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.661+0000 7fa1d2ffd640 10 monclient: get_auth_request con 0x7fa1c006de00 auth_method 0 2024-03-25T19:43:59.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.662+0000 7fa1bbfff640 10 client.4634.objecter ms_handle_connect 0x7fa1c006de00 2024-03-25T19:43:59.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fa1c8054810 con 0x7fa1d413f710 2024-03-25T19:43:59.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 10 client.4634.objecter ms_dispatch 0x7fa1d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:59.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:59.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:59.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T19:43:59.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1c8050320 con 0x7fa1d413f710 2024-03-25T19:43:59.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:59.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.667+0000 7fa1bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:59.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.892+0000 7fa1d9939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:59.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.892+0000 7fa1d9939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:59.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.892+0000 7fa1d9939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:43:59.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.892+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1d413b5c0 con 0x7fa1d413f710 2024-03-25T19:43:59.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.936+0000 7fa1bbfff640 1 -- 172.21.15.47:0/4266540998 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fa1d413b5c0 con 0x7fa1d413f710 2024-03-25T19:43:59.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.936+0000 7fa1bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:59.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.936+0000 7fa1bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:59.936 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:43:59.936 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:59.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 10 librados: watch_flush enter 2024-03-25T19:43:59.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 10 librados: watch_flush exit 2024-03-25T19:43:59.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 10 client.4634.objecter successfully canceled tick 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 >> v1:172.21.15.142:6800/1025302268 conn(0x7fa1c006de00 legacy=0x7fa1c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 10 monclient: shutdown 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 >> v1:172.21.15.47:6790/0 conn(0x7fa1d413f710 legacy=0x7fa1d41dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 shutdown_connections 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.938+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 >> 172.21.15.47:0/4266540998 conn(0x7fa1d40b7ea0 msgr2=0x7fa1d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.939+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 shutdown_connections 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.939+0000 7fa1d9939640 1 -- 172.21.15.47:0/4266540998 wait complete. 2024-03-25T19:43:59.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:43:59.939+0000 7fa1d9939640 1 librados: shutdown 2024-03-25T19:43:59.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:03.812 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:41.048 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.820 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:41.055 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.858 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:41.048 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.860 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:41.050 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.867 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:41.103 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.871 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:41.107 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.883 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:41.118 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.896 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:09:41.086 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:03.902 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:41.137 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:06.757 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.871+0000 7febbcdc8640 10 monclient: get_monmap_and_config 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.871+0000 7febbcdc8640 10 monclient: build_initial_monmap 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.871+0000 7febbcdc8640 10 monclient: monmap: 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:06.872155+0000 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:06.872155+0000 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:06.871 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:06.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.872+0000 7febbcdc8640 1 Processor -- start 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.872+0000 7febbcdc8640 1 -- start start 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.872+0000 7febbcdc8640 10 monclient: init 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient(hunting): picked mon.noname-b con 0x7febb80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient(hunting): picked mon.noname-a con 0x7febb813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient(hunting): picked mon.noname-c con 0x7febb81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb8142520 con 0x7febb813f710 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80b28a0 con 0x7febb81430c0 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80b2a30 con 0x7febb80a98f0 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:06.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.873+0000 7febbcdc8640 10 monclient(hunting): authenticate will time out at 6162.159253s 2024-03-25T19:44:06.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.890+0000 7febb659c640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7febb80a98f0 0x7febb80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51900/0 (socket says 172.21.15.47:51900) 2024-03-25T19:44:06.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.890+0000 7febb659c640 1 -- 172.21.15.47:0/2647725895 learned_addr learned my addr 172.21.15.47:0/2647725895 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:06.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.907+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085800164 0 0) 0x7febb8142520 con 0x7febb813f710 2024-03-25T19:44:06.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.907+0000 7febb559a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:06.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.907+0000 7febb559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:06.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.907+0000 7febb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:06.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.907+0000 7febb559a640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:44:06.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.907+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba0003520 con 0x7febb813f710 2024-03-25T19:44:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.915+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492124642 0 0) 0x7febb80b2a30 con 0x7febb80a98f0 2024-03-25T19:44:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.915+0000 7febb559a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.915+0000 7febb559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.915+0000 7febb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.915+0000 7febb559a640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:44:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.915+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb8142520 con 0x7febb80a98f0 2024-03-25T19:44:06.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.917+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294271634 0 0) 0x7febb80b28a0 con 0x7febb81430c0 2024-03-25T19:44:06.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.917+0000 7febb559a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:06.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.917+0000 7febb559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:06.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.917+0000 7febb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:06.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.917+0000 7febb559a640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:44:06.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.917+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb80b2a30 con 0x7febb81430c0 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623766999 0 0) 0x7feba0003520 con 0x7febb813f710 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 >> v1:172.21.15.47:6790/0 conn(0x7febb80a98f0 legacy=0x7febb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 >> v1:172.21.15.142:6789/0 conn(0x7febb81430c0 legacy=0x7febb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 1 monclient: found mon.noname-a 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb80b44c0 con 0x7febb813f710 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: _finish_auth 0 2024-03-25T19:44:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7febac001690 con 0x7febb813f710 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febb559a640 10 monclient: dump: 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:06.924 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febbcdc8640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:44:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.923+0000 7febbcdc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:06.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febac0017f0 con 0x7febb813f710 2024-03-25T19:44:06.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febb559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:06.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febb559a640 1 -- 172.21.15.47:0/2647725895 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7febac001c00 con 0x7febb813f710 2024-03-25T19:44:06.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:06.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febb559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:06.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febb559a640 10 monclient: dump: 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febbcdc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:06.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.938+0000 7febbcdc8640 4 set_mon_vals no callback set 2024-03-25T19:44:06.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.939+0000 7febbcdc8640 10 monclient: shutdown 2024-03-25T19:44:06.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.939+0000 7febbcdc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.939+0000 7febbcdc8640 1 -- 172.21.15.47:0/2647725895 >> v1:172.21.15.47:6789/0 conn(0x7febb813f710 legacy=0x7febb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.939+0000 7febbcdc8640 1 -- 172.21.15.47:0/2647725895 shutdown_connections 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.939+0000 7febbcdc8640 1 -- 172.21.15.47:0/2647725895 >> 172.21.15.47:0/2647725895 conn(0x7febb80b7ea0 msgr2=0x7febb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.940+0000 7febbcdc8640 1 -- 172.21.15.47:0/2647725895 shutdown_connections 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.940+0000 7febbcdc8640 1 -- 172.21.15.47:0/2647725895 wait complete. 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.940+0000 7febbcdc8640 10 monclient: build_initial_monmap 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 10 monclient: monmap: 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:06.942 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 1 librados: starting msgr at 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 1 librados: starting objecter 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 1 Processor -- start 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 1 -- start start 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 1 librados: setting wanted keys 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 1 librados: calling monclient init 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 10 monclient: init 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.941+0000 7febbcdc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 10 monclient(hunting): picked mon.noname-b con 0x7febb80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:06.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 10 monclient(hunting): picked mon.noname-a con 0x7febb813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:06.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 10 monclient(hunting): picked mon.noname-c con 0x7febb81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:06.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb81d51f0 con 0x7febb813f710 2024-03-25T19:44:06.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb81379b0 con 0x7febb80a98f0 2024-03-25T19:44:06.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb81db8b0 con 0x7febb81430c0 2024-03-25T19:44:06.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:06.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.942+0000 7febbcdc8640 10 monclient(hunting): authenticate will time out at 6162.228006s 2024-03-25T19:44:06.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.956+0000 7febb5d9b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7febb813f710 0x7febb81daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33600/0 (socket says 172.21.15.47:33600) 2024-03-25T19:44:06.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.956+0000 7febb5d9b640 1 -- 172.21.15.47:0/3525154287 learned_addr learned my addr 172.21.15.47:0/3525154287 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:06.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.957+0000 7feba6ffd640 10 client.?.objecter ms_handle_connect 0x7febb813f710 2024-03-25T19:44:06.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.957+0000 7feba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:06.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.961+0000 7feba6ffd640 10 client.?.objecter ms_handle_connect 0x7febb81430c0 2024-03-25T19:44:06.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.961+0000 7feba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:06.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.970+0000 7feba6ffd640 10 client.?.objecter ms_handle_connect 0x7febb80a98f0 2024-03-25T19:44:06.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.970+0000 7feba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:06.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.980+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697906707 0 0) 0x7febb81d51f0 con 0x7febb813f710 2024-03-25T19:44:06.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.980+0000 7feba6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:06.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.980+0000 7feba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:06.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.980+0000 7feba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:06.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.980+0000 7feba6ffd640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:44:06.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.980+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98004eb0 con 0x7febb813f710 2024-03-25T19:44:06.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.982+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277353240 0 0) 0x7febb81db8b0 con 0x7febb81430c0 2024-03-25T19:44:06.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.982+0000 7feba6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:06.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.982+0000 7feba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:06.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.982+0000 7feba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:06.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.982+0000 7feba6ffd640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:44:06.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.982+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb81d51f0 con 0x7febb81430c0 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 909092371 0 0) 0x7feb98004eb0 con 0x7febb813f710 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb81db8b0 con 0x7febb813f710 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7febac001bc0 con 0x7febb813f710 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:06.995 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:06.996 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:06.996 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:06.996 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.995+0000 7feba6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:06.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.999+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284921191 0 0) 0x7febb81379b0 con 0x7febb80a98f0 2024-03-25T19:44:06.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.999+0000 7feba6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:06.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.999+0000 7feba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:06.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.999+0000 7feba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:06.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.999+0000 7feba6ffd640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:44:06.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:06.999+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98004eb0 con 0x7febb80a98f0 2024-03-25T19:44:07.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.007+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 798031899 0 0) 0x7febb81d51f0 con 0x7febb81430c0 2024-03-25T19:44:07.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.007+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb81379b0 con 0x7febb81430c0 2024-03-25T19:44:07.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.007+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7febb0001b50 con 0x7febb81430c0 2024-03-25T19:44:07.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.007+0000 7feba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.007+0000 7feba6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.007+0000 7feba6ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2461863059 0 0) 0x7febb81db8b0 con 0x7febb813f710 2024-03-25T19:44:07.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 >> v1:172.21.15.47:6790/0 conn(0x7febb81430c0 legacy=0x7febb81db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 >> v1:172.21.15.142:6789/0 conn(0x7febb80a98f0 legacy=0x7febb8138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:07.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 1 monclient: found mon.a 2024-03-25T19:44:07.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:07.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:07.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb81e5e00 con 0x7febb813f710 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7feba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 5 monclient: authenticate success, global_id 4650 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 10 monclient: _renew_subs 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb81e2dd0 con 0x7febb813f710 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 10 client.4650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 10 monclient: _renew_subs 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febb81e32b0 con 0x7febb813f710 2024-03-25T19:44:07.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.008+0000 7febbcdc8640 1 librados: init done 2024-03-25T19:44:07.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.009+0000 7febbcdc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.009+0000 7febbcdc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.009+0000 7febbcdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:07.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.009+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb8137b40 con 0x7febb813f710 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febac0011f0 con 0x7febb813f710 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7feba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7febac003b00 con 0x7febb813f710 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7feba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7feba6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7feba6ffd640 10 monclient: dump: 2024-03-25T19:44:07.331 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:07.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.331+0000 7febb4d99640 4 set_mon_vals no callback set 2024-03-25T19:44:07.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.332+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7febac003d20 con 0x7febb813f710 2024-03-25T19:44:07.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.333+0000 7febb659c640 10 monclient: get_auth_request con 0x7feb9806dfb0 auth_method 0 2024-03-25T19:44:07.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.333+0000 7feba6ffd640 10 client.4650.objecter ms_handle_connect 0x7feb9806dfb0 2024-03-25T19:44:07.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.343+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7febac0548d0 con 0x7febb813f710 2024-03-25T19:44:07.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.343+0000 7feba6ffd640 10 client.4650.objecter ms_dispatch 0x7febb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:07.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.343+0000 7feba6ffd640 3 client.4650.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:07.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.343+0000 7feba6ffd640 3 client.4650.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:07.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.344+0000 7feba6ffd640 20 client.4650.objecter dump_active .. 0 homeless 2024-03-25T19:44:07.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.344+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febac0503e0 con 0x7febb813f710 2024-03-25T19:44:07.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.344+0000 7feba6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.344+0000 7feba6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:07.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.570+0000 7febbcdc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:07.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.570+0000 7febbcdc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:07.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.570+0000 7febbcdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:07.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.570+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febb8074cc0 con 0x7febb813f710 2024-03-25T19:44:07.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.598+0000 7feba6ffd640 1 -- 172.21.15.47:0/3525154287 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7febac052310 con 0x7febb813f710 2024-03-25T19:44:07.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.598+0000 7feba6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:07.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.598+0000 7feba6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:07.598 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:07.598 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 10 librados: watch_flush enter 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 10 librados: watch_flush exit 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 20 client.4650.objecter shutdown clearing up homeless session... 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 10 client.4650.objecter successfully canceled tick 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 >> v1:172.21.15.142:6800/1025302268 conn(0x7feb9806dfb0 legacy=0x7feb98070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 10 monclient: shutdown 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 >> v1:172.21.15.47:6789/0 conn(0x7febb813f710 legacy=0x7febb81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 shutdown_connections 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 >> 172.21.15.47:0/3525154287 conn(0x7febb80b7ea0 msgr2=0x7febb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 shutdown_connections 2024-03-25T19:44:07.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 -- 172.21.15.47:0/3525154287 wait complete. 2024-03-25T19:44:07.602 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:07.601+0000 7febbcdc8640 1 librados: shutdown 2024-03-25T19:44:07.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:14.419 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:14.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.524+0000 7fe8e677b640 10 monclient: get_monmap_and_config 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.524+0000 7fe8e677b640 10 monclient: build_initial_monmap 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.524+0000 7fe8e677b640 10 monclient: monmap: 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:14.525140+0000 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:14.525140+0000 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:14.524 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 1 Processor -- start 2024-03-25T19:44:14.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 1 -- start start 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient: init 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e00b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8e00b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e014be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e00af940 con 0x7fe8e014be90 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e00b0a70 con 0x7fe8e00b4d60 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e00b0d50 con 0x7fe8e00b1730 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:14.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.525+0000 7fe8e677b640 10 monclient(hunting): authenticate will time out at 6169.811926s 2024-03-25T19:44:14.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.530+0000 7fe8e4d18640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe8e014be90 0x7fe8e014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53642/0 (socket says 172.21.15.47:53642) 2024-03-25T19:44:14.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.530+0000 7fe8e4d18640 1 -- 172.21.15.47:0/2037536234 learned_addr learned my addr 172.21.15.47:0/2037536234 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:14.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.544+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293937280 0 0) 0x7fe8e00b0a70 con 0x7fe8e00b4d60 2024-03-25T19:44:14.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.544+0000 7fe8deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.544+0000 7fe8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.544+0000 7fe8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.544+0000 7fe8deffd640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:44:14.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.544+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c8003520 con 0x7fe8e00b4d60 2024-03-25T19:44:14.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.549+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469130551 0 0) 0x7fe8e00af940 con 0x7fe8e014be90 2024-03-25T19:44:14.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.549+0000 7fe8deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.549+0000 7fe8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.549+0000 7fe8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.549+0000 7fe8deffd640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:44:14.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.549+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e00b0a70 con 0x7fe8e014be90 2024-03-25T19:44:14.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.558+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644849725 0 0) 0x7fe8e00b0d50 con 0x7fe8e00b1730 2024-03-25T19:44:14.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.558+0000 7fe8deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.558+0000 7fe8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.558+0000 7fe8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.558+0000 7fe8deffd640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:44:14.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.558+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e00af940 con 0x7fe8e00b1730 2024-03-25T19:44:14.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002446566 0 0) 0x7fe8e00af940 con 0x7fe8e00b1730 2024-03-25T19:44:14.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 >> v1:172.21.15.142:6789/0 conn(0x7fe8e00b4d60 legacy=0x7fe8e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 >> v1:172.21.15.47:6789/0 conn(0x7fe8e014be90 legacy=0x7fe8e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:14.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 1 monclient: found mon.noname-b 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00b1030 con 0x7fe8e00b1730 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.568+0000 7fe8deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8e677b640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8e677b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe8d40014b0 con 0x7fe8e00b1730 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8deffd640 10 monclient: dump: 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:14.569 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.569+0000 7fe8e677b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.578+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8d4001780 con 0x7fe8e00b1730 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.578+0000 7fe8deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.578+0000 7fe8deffd640 1 -- 172.21.15.47:0/2037536234 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe8d4001b10 con 0x7fe8e00b1730 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.578+0000 7fe8deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.578+0000 7fe8deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.578+0000 7fe8deffd640 10 monclient: dump: 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.578 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:14.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 4 set_mon_vals no callback set 2024-03-25T19:44:14.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 10 monclient: shutdown 2024-03-25T19:44:14.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:14.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 1 -- 172.21.15.47:0/2037536234 >> v1:172.21.15.47:6790/0 conn(0x7fe8e00b1730 legacy=0x7fe8e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 1 -- 172.21.15.47:0/2037536234 shutdown_connections 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.579+0000 7fe8e677b640 1 -- 172.21.15.47:0/2037536234 >> 172.21.15.47:0/2037536234 conn(0x7fe8e00617a0 msgr2=0x7fe8e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.580+0000 7fe8e677b640 1 -- 172.21.15.47:0/2037536234 shutdown_connections 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.580+0000 7fe8e677b640 1 -- 172.21.15.47:0/2037536234 wait complete. 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.580+0000 7fe8e677b640 10 monclient: build_initial_monmap 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.580+0000 7fe8e677b640 10 monclient: monmap: 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:14.581 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.580+0000 7fe8e677b640 1 librados: starting msgr at 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.580+0000 7fe8e677b640 1 librados: starting objecter 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 1 Processor -- start 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 1 -- start start 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 1 librados: setting wanted keys 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 1 librados: calling monclient init 2024-03-25T19:44:14.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 10 monclient: init 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.581+0000 7fe8e677b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8e00b1730 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e00b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e014be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e013bc80 con 0x7fe8e00b4d60 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e013aec0 con 0x7fe8e00b1730 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e013b860 con 0x7fe8e014be90 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:14.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.582+0000 7fe8e677b640 10 monclient(hunting): authenticate will time out at 6169.868104s 2024-03-25T19:44:14.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.596+0000 7fe8e4d18640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe8e014be90 0x7fe8e01daea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43784/0 (socket says 172.21.15.47:43784) 2024-03-25T19:44:14.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.596+0000 7fe8e4d18640 1 -- 172.21.15.47:0/3913417616 learned_addr learned my addr 172.21.15.47:0/3913417616 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:14.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.596+0000 7fe8dcff9640 10 client.?.objecter ms_handle_connect 0x7fe8e014be90 2024-03-25T19:44:14.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.596+0000 7fe8dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:14.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.599+0000 7fe8dcff9640 10 client.?.objecter ms_handle_connect 0x7fe8e00b1730 2024-03-25T19:44:14.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.599+0000 7fe8dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:14.602 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.603+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157885067 0 0) 0x7fe8e013aec0 con 0x7fe8e00b1730 2024-03-25T19:44:14.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.603+0000 7fe8dcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.603+0000 7fe8dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.603+0000 7fe8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.603+0000 7fe8dcff9640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:44:14.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.603+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8cc004eb0 con 0x7fe8e00b1730 2024-03-25T19:44:14.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.607+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283969272 0 0) 0x7fe8e013b860 con 0x7fe8e014be90 2024-03-25T19:44:14.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.607+0000 7fe8dcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:14.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.607+0000 7fe8dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:14.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.607+0000 7fe8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:14.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.607+0000 7fe8dcff9640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:44:14.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.607+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e013aec0 con 0x7fe8e014be90 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1037664206 0 0) 0x7fe8cc004eb0 con 0x7fe8e00b1730 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8e013b860 con 0x7fe8e00b1730 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe8d0001950 con 0x7fe8e00b1730 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 10 monclient(hunting): dump: 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.612 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.612+0000 7fe8dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:14.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.613+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1568964477 0 0) 0x7fe8e013aec0 con 0x7fe8e014be90 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.613+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8cc004eb0 con 0x7fe8e014be90 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.613+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe8d8001b70 con 0x7fe8e014be90 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.613+0000 7fe8dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.613+0000 7fe8dcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.613+0000 7fe8dcff9640 10 monclient(hunting): dump: 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:14.614 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:14.615 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.615+0000 7fe8dcff9640 10 client.?.objecter ms_handle_connect 0x7fe8e00b4d60 2024-03-25T19:44:14.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.615+0000 7fe8dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2694448200 0 0) 0x7fe8e013b860 con 0x7fe8e00b1730 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 >> v1:172.21.15.47:6790/0 conn(0x7fe8e014be90 legacy=0x7fe8e01daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 >> v1:172.21.15.47:6789/0 conn(0x7fe8e00b4d60 legacy=0x7fe8e0137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 1 monclient: found mon.b 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e01378a0 con 0x7fe8e00b1730 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:14.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 10 monclient: _renew_subs 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e0142860 con 0x7fe8e00b1730 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 10 monclient: _renew_subs 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e01db5b0 con 0x7fe8e00b1730 2024-03-25T19:44:14.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.626+0000 7fe8e677b640 1 librados: init done 2024-03-25T19:44:14.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.628+0000 7fe8e677b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:14.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.628+0000 7fe8e677b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:14.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.628+0000 7fe8e677b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:14.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.628+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e00627b0 con 0x7fe8e00b1730 2024-03-25T19:44:14.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.905+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8d00035f0 con 0x7fe8e00b1730 2024-03-25T19:44:14.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.905+0000 7fe8dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:14.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.906+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fe8d0003900 con 0x7fe8e00b1730 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.906+0000 7fe8dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.906+0000 7fe8dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.906+0000 7fe8dcff9640 10 monclient: dump: 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.907 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:14.908 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:14.908 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:14.908 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:14.908 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:14.908 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:14.908 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:14.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.906+0000 7fe8de7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:14.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.906+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fe8d00198d0 con 0x7fe8e00b1730 2024-03-25T19:44:14.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.907+0000 7fe8df7fe640 10 monclient: get_auth_request con 0x7fe8cc06de80 auth_method 0 2024-03-25T19:44:14.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.907+0000 7fe8dcff9640 10 client.4657.objecter ms_handle_connect 0x7fe8cc06de80 2024-03-25T19:44:14.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.911+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fe8d0053570 con 0x7fe8e00b1730 2024-03-25T19:44:14.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.911+0000 7fe8dcff9640 10 client.4657.objecter ms_dispatch 0x7fe8e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:14.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.911+0000 7fe8dcff9640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:14.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.911+0000 7fe8dcff9640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:14.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.911+0000 7fe8dcff9640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T19:44:14.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.912+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8d004eeb0 con 0x7fe8e00b1730 2024-03-25T19:44:14.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.912+0000 7fe8dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:14.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:14.912+0000 7fe8dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.173+0000 7fe8e677b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.173+0000 7fe8e677b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.173+0000 7fe8e677b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:15.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.173+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8e00ae330 con 0x7fe8e00b1730 2024-03-25T19:44:15.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.200+0000 7fe8dcff9640 1 -- 172.21.15.47:0/3913417616 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fe8d0050fb0 con 0x7fe8e00b1730 2024-03-25T19:44:15.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.200+0000 7fe8dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:15.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.200+0000 7fe8dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:15.200 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:15.200 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:15.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 10 librados: watch_flush enter 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 10 librados: watch_flush exit 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 10 client.4657.objecter successfully canceled tick 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 >> v1:172.21.15.142:6800/1025302268 conn(0x7fe8cc06de80 legacy=0x7fe8cc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 10 monclient: shutdown 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 >> v1:172.21.15.142:6789/0 conn(0x7fe8e00b1730 legacy=0x7fe8e013b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 shutdown_connections 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.202+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 >> 172.21.15.47:0/3913417616 conn(0x7fe8e00617a0 msgr2=0x7fe8e00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.203+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 shutdown_connections 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.203+0000 7fe8e677b640 1 -- 172.21.15.47:0/3913417616 wait complete. 2024-03-25T19:44:15.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:15.203+0000 7fe8e677b640 1 librados: shutdown 2024-03-25T19:44:15.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:22.020 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.138+0000 7f98dbcf4640 10 monclient: get_monmap_and_config 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.138+0000 7f98dbcf4640 10 monclient: build_initial_monmap 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.138+0000 7f98dbcf4640 10 monclient: monmap: 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:22.139220+0000 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:22.139220+0000 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:22.138 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:22.139 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:22.139 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:22.139 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 1 Processor -- start 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 1 -- start start 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 10 monclient: init 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 10 monclient(hunting): picked mon.noname-b con 0x7f98d40a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.139+0000 7f98dbcf4640 10 monclient(hunting): picked mon.noname-c con 0x7f98d413f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.140+0000 7f98dbcf4640 10 monclient(hunting): picked mon.noname-a con 0x7f98d41430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:22.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.140+0000 7f98dbcf4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d4142520 con 0x7f98d41430c0 2024-03-25T19:44:22.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.140+0000 7f98dbcf4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d40b28a0 con 0x7f98d413f710 2024-03-25T19:44:22.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.140+0000 7f98dbcf4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d40b2a30 con 0x7f98d40a98f0 2024-03-25T19:44:22.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.140+0000 7f98dbcf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:22.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.140+0000 7f98dbcf4640 10 monclient(hunting): authenticate will time out at 6177.426089s 2024-03-25T19:44:22.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.160+0000 7f98d928f640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f98d413f710 0x7f98d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:58334/0 (socket says 172.21.15.47:58334) 2024-03-25T19:44:22.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.161+0000 7f98d928f640 1 -- 172.21.15.47:0/3972209397 learned_addr learned my addr 172.21.15.47:0/3972209397 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:22.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.171+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641698058 0 0) 0x7f98d40b28a0 con 0x7f98d413f710 2024-03-25T19:44:22.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.171+0000 7f98d8a8e640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.171+0000 7f98d8a8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.171+0000 7f98d8a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.171+0000 7f98d8a8e640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:44:22.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.171+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c0003520 con 0x7f98d413f710 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1497428514 0 0) 0x7f98c0003520 con 0x7f98d413f710 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 >> v1:172.21.15.47:6790/0 conn(0x7f98d40a98f0 legacy=0x7f98d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 >> v1:172.21.15.47:6789/0 conn(0x7f98d41430c0 legacy=0x7f98d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: _finish_hunting 0 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 1 monclient: found mon.noname-c 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d40b44c0 con 0x7f98d413f710 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: _finish_auth 0 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: _check_auth_tickets 2024-03-25T19:44:22.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f98c4001690 con 0x7f98d413f710 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98d8a8e640 10 monclient: dump: 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:22.190 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98dbcf4640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:44:22.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.189+0000 7f98dbcf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c40017f0 con 0x7f98d413f710 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98d8a8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98d8a8e640 1 -- 172.21.15.47:0/3972209397 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f98c4001c40 con 0x7f98d413f710 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98dbcf4640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98dbcf4640 4 set_mon_vals no callback set 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98d8a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98d8a8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.193+0000 7f98d8a8e640 10 monclient: dump: 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:22.194 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 10 monclient: shutdown 2024-03-25T19:44:22.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:22.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 1 -- 172.21.15.47:0/3972209397 >> v1:172.21.15.142:6789/0 conn(0x7f98d413f710 legacy=0x7f98d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 1 -- 172.21.15.47:0/3972209397 shutdown_connections 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 1 -- 172.21.15.47:0/3972209397 >> 172.21.15.47:0/3972209397 conn(0x7f98d40b7ea0 msgr2=0x7f98d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 1 -- 172.21.15.47:0/3972209397 shutdown_connections 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.194+0000 7f98dbcf4640 1 -- 172.21.15.47:0/3972209397 wait complete. 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 10 monclient: build_initial_monmap 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 10 monclient: monmap: 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:22.197 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 1 librados: starting msgr at 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 1 librados: starting objecter 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 1 Processor -- start 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 1 -- start start 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 1 librados: setting wanted keys 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 1 librados: calling monclient init 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.195+0000 7f98dbcf4640 10 monclient: init 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient(hunting): picked mon.noname-b con 0x7f98d40a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient(hunting): picked mon.noname-a con 0x7f98d413f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient(hunting): picked mon.noname-c con 0x7f98d41430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d41370a0 con 0x7f98d413f710 2024-03-25T19:44:22.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d4135090 con 0x7f98d40a98f0 2024-03-25T19:44:22.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d4135220 con 0x7f98d41430c0 2024-03-25T19:44:22.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:22.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.196+0000 7f98dbcf4640 10 monclient(hunting): authenticate will time out at 6177.482603s 2024-03-25T19:44:22.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.202+0000 7f98d9a90640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f98d40a98f0 0x7f98d41dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:58364/0 (socket says 172.21.15.47:58364) 2024-03-25T19:44:22.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.202+0000 7f98d9a90640 1 -- 172.21.15.47:0/384527971 learned_addr learned my addr 172.21.15.47:0/384527971 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:22.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.205+0000 7f98ca7fc640 10 client.?.objecter ms_handle_connect 0x7f98d40a98f0 2024-03-25T19:44:22.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.205+0000 7f98ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:22.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.210+0000 7f98ca7fc640 10 client.?.objecter ms_handle_connect 0x7f98d41430c0 2024-03-25T19:44:22.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.210+0000 7f98ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:22.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.217+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143874763 0 0) 0x7f98d4135220 con 0x7f98d41430c0 2024-03-25T19:44:22.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.217+0000 7f98ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.217+0000 7f98ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.217+0000 7f98ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.217+0000 7f98ca7fc640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:44:22.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.217+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b8004eb0 con 0x7f98d41430c0 2024-03-25T19:44:22.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.221+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707861663 0 0) 0x7f98d4135090 con 0x7f98d40a98f0 2024-03-25T19:44:22.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.221+0000 7f98ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.221+0000 7f98ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.221+0000 7f98ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.221+0000 7f98ca7fc640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:44:22.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.221+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d4135220 con 0x7f98d40a98f0 2024-03-25T19:44:22.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.225+0000 7f98ca7fc640 10 client.?.objecter ms_handle_connect 0x7f98d413f710 2024-03-25T19:44:22.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.225+0000 7f98ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:22.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.233+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719997699 0 0) 0x7f98d41370a0 con 0x7f98d413f710 2024-03-25T19:44:22.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.233+0000 7f98ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:22.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.233+0000 7f98ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:22.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.233+0000 7f98ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:22.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.233+0000 7f98ca7fc640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:44:22.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.233+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d4135090 con 0x7f98d413f710 2024-03-25T19:44:22.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2043574323 0 0) 0x7f98d4135220 con 0x7f98d40a98f0 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d41370a0 con 0x7f98d40a98f0 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f98d00019f0 con 0x7f98d40a98f0 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:22.234 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:22.235 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:22.235 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.234+0000 7f98ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:22.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.237+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840505011 0 0) 0x7f98b8004eb0 con 0x7f98d41430c0 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.237+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d4135220 con 0x7f98d41430c0 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.237+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f98cc001b20 con 0x7f98d41430c0 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.237+0000 7f98ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.237+0000 7f98ca7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.237+0000 7f98ca7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:22.237 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1184771423 0 0) 0x7f98d41370a0 con 0x7f98d40a98f0 2024-03-25T19:44:22.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 >> v1:172.21.15.47:6790/0 conn(0x7f98d41430c0 legacy=0x7f98d41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 >> v1:172.21.15.47:6789/0 conn(0x7f98d413f710 legacy=0x7f98d41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 1 monclient: found mon.b 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d41d5000 con 0x7f98d40a98f0 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 10 monclient: _renew_subs 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d41d52b0 con 0x7f98d40a98f0 2024-03-25T19:44:22.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:22.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 10 monclient: _renew_subs 2024-03-25T19:44:22.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:22.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d4135910 con 0x7f98d40a98f0 2024-03-25T19:44:22.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.241+0000 7f98dbcf4640 1 librados: init done 2024-03-25T19:44:22.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.242+0000 7f98dbcf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:22.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.242+0000 7f98dbcf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:22.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.242+0000 7f98dbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:22.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.242+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d40627b0 con 0x7f98d40a98f0 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d0003610 con 0x7f98d40a98f0 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f98d00039e0 con 0x7f98d40a98f0 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98ca7fc640 10 monclient: dump: 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.456 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:22.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.456+0000 7f98cbfff640 4 set_mon_vals no callback set 2024-03-25T19:44:22.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.516+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f98d0003b40 con 0x7f98d40a98f0 2024-03-25T19:44:22.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.517+0000 7f98d928f640 10 monclient: get_auth_request con 0x7f98b806df70 auth_method 0 2024-03-25T19:44:22.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.517+0000 7f98ca7fc640 10 client.4663.objecter ms_handle_connect 0x7f98b806df70 2024-03-25T19:44:22.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.518+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f98d0054860 con 0x7f98d40a98f0 2024-03-25T19:44:22.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.518+0000 7f98ca7fc640 10 client.4663.objecter ms_dispatch 0x7f98d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:22.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.518+0000 7f98ca7fc640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:22.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.518+0000 7f98ca7fc640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:22.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.518+0000 7f98ca7fc640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:44:22.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.524+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98d00501a0 con 0x7f98d40a98f0 2024-03-25T19:44:22.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.524+0000 7f98ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:22.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.524+0000 7f98ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:22.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.741+0000 7f98dbcf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:22.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.741+0000 7f98dbcf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:22.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.741+0000 7f98dbcf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:44:22.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.741+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98d413bfe0 con 0x7f98d40a98f0 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.833+0000 7f98ca7fc640 1 -- 172.21.15.47:0/384527971 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f98d00522a0 con 0x7f98d40a98f0 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.833+0000 7f98ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.833+0000 7f98ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:22.833 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 10 librados: watch_flush enter 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 10 librados: watch_flush exit 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 >> v1:172.21.15.142:6800/1025302268 conn(0x7f98b806df70 legacy=0x7f98b8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 10 monclient: shutdown 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:22.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 >> v1:172.21.15.142:6789/0 conn(0x7f98d40a98f0 legacy=0x7f98d41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:22.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 shutdown_connections 2024-03-25T19:44:22.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 >> 172.21.15.47:0/384527971 conn(0x7f98d40b7ea0 msgr2=0x7f98d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:22.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 shutdown_connections 2024-03-25T19:44:22.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.836+0000 7f98dbcf4640 1 -- 172.21.15.47:0/384527971 wait complete. 2024-03-25T19:44:22.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:22.837+0000 7f98dbcf4640 1 librados: shutdown 2024-03-25T19:44:22.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:23.880 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:01.116 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:23.928 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:01.163 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:23.956 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:01.191 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:23.996 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:01.232 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:24.000 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:01.190 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:24.004 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:01.194 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:24.013 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:01.249 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:24.029 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:01.219 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:24.043 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:01.279 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:29.654 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.770+0000 7f65dfc5c640 10 monclient: get_monmap_and_config 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.770+0000 7f65dfc5c640 10 monclient: build_initial_monmap 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.770+0000 7f65dfc5c640 10 monclient: monmap: 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:29.772204+0000 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:29.772204+0000 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:29.771 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:29.772 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:29.772 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:29.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.771+0000 7f65dfc5c640 1 Processor -- start 2024-03-25T19:44:29.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.771+0000 7f65dfc5c640 1 -- start start 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.771+0000 7f65dfc5c640 10 monclient: init 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient(hunting): picked mon.noname-a con 0x7f65d80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient(hunting): picked mon.noname-b con 0x7f65d813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient(hunting): picked mon.noname-c con 0x7f65d81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d8142520 con 0x7f65d80a98f0 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d80b28a0 con 0x7f65d81430c0 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d80b2a30 con 0x7f65d813f710 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:29.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.772+0000 7f65dfc5c640 10 monclient(hunting): authenticate will time out at 6185.059328s 2024-03-25T19:44:29.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.775+0000 7f65de1f9640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f65d81430c0 0x7f65d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:59318/0 (socket says 172.21.15.47:59318) 2024-03-25T19:44:29.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.775+0000 7f65de1f9640 1 -- 172.21.15.47:0/4051520323 learned_addr learned my addr 172.21.15.47:0/4051520323 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:29.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.784+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443762539 0 0) 0x7f65d80b28a0 con 0x7f65d81430c0 2024-03-25T19:44:29.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.784+0000 7f65dc9f6640 10 monclient(hunting): _negotiate 2024-03-25T19:44:29.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.784+0000 7f65dc9f6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:29.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.784+0000 7f65dc9f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:29.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.784+0000 7f65dc9f6640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:44:29.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.784+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c4003520 con 0x7f65d81430c0 2024-03-25T19:44:29.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.792+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280556489 0 0) 0x7f65d8142520 con 0x7f65d80a98f0 2024-03-25T19:44:29.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.792+0000 7f65dc9f6640 10 monclient(hunting): _negotiate 2024-03-25T19:44:29.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.792+0000 7f65dc9f6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:29.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.792+0000 7f65dc9f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:29.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.792+0000 7f65dc9f6640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:44:29.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.792+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d80b28a0 con 0x7f65d80a98f0 2024-03-25T19:44:29.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.806+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357243369 0 0) 0x7f65d80b2a30 con 0x7f65d813f710 2024-03-25T19:44:29.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.806+0000 7f65dc9f6640 10 monclient(hunting): _negotiate 2024-03-25T19:44:29.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.806+0000 7f65dc9f6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:29.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.806+0000 7f65dc9f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:29.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.806+0000 7f65dc9f6640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:44:29.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.806+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d8142520 con 0x7f65d813f710 2024-03-25T19:44:29.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2655803381 0 0) 0x7f65d80b28a0 con 0x7f65d80a98f0 2024-03-25T19:44:29.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 >> v1:172.21.15.47:6790/0 conn(0x7f65d813f710 legacy=0x7f65d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 >> v1:172.21.15.142:6789/0 conn(0x7f65d81430c0 legacy=0x7f65d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: _finish_hunting 0 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 1 monclient: found mon.noname-a 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d80b44c0 con 0x7f65d80a98f0 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: _finish_auth 0 2024-03-25T19:44:29.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: _check_auth_tickets 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f65d4001670 con 0x7f65d80a98f0 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dfc5c640 5 monclient: authenticate success, global_id 4665 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dfc5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dc9f6640 10 monclient: dump: 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:29.818 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:29.819 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:29.819 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:29.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.814+0000 7f65dfc5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65d40017d0 con 0x7f65d80a98f0 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dc9f6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dc9f6640 1 -- 172.21.15.47:0/4051520323 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f65d4001be0 con 0x7f65d80a98f0 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dc9f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dc9f6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dc9f6640 10 monclient: dump: 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:29.827 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dfc5c640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:29.828 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.826+0000 7f65dfc5c640 4 set_mon_vals no callback set 2024-03-25T19:44:29.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 10 monclient: shutdown 2024-03-25T19:44:29.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:29.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4051520323 >> v1:172.21.15.47:6789/0 conn(0x7f65d80a98f0 legacy=0x7f65d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4051520323 shutdown_connections 2024-03-25T19:44:29.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4051520323 >> 172.21.15.47:0/4051520323 conn(0x7f65d80b7ea0 msgr2=0x7f65d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:29.829 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4051520323 shutdown_connections 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.827+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4051520323 wait complete. 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.828+0000 7f65dfc5c640 10 monclient: build_initial_monmap 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.828+0000 7f65dfc5c640 10 monclient: monmap: 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 1 librados: starting msgr at 2024-03-25T19:44:29.830 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 1 librados: starting objecter 2024-03-25T19:44:29.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 1 Processor -- start 2024-03-25T19:44:29.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 1 -- start start 2024-03-25T19:44:29.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 1 librados: setting wanted keys 2024-03-25T19:44:29.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 1 librados: calling monclient init 2024-03-25T19:44:29.831 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 10 monclient: init 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.829+0000 7f65dfc5c640 10 monclient(hunting): picked mon.noname-b con 0x7f65d80a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 10 monclient(hunting): picked mon.noname-c con 0x7f65d813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 10 monclient(hunting): picked mon.noname-a con 0x7f65d81430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d8139720 con 0x7f65d81430c0 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d8138af0 con 0x7f65d80a98f0 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d8138d60 con 0x7f65d813f710 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:29.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.830+0000 7f65dfc5c640 10 monclient(hunting): authenticate will time out at 6185.117049s 2024-03-25T19:44:29.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.843+0000 7f65de1f9640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f65d81430c0 0x7f65d80b0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34206/0 (socket says 172.21.15.47:34206) 2024-03-25T19:44:29.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.843+0000 7f65de1f9640 1 -- 172.21.15.47:0/4188570079 learned_addr learned my addr 172.21.15.47:0/4188570079 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:29.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.844+0000 7f65ce7fc640 10 client.?.objecter ms_handle_connect 0x7f65d81430c0 2024-03-25T19:44:29.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.844+0000 7f65ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:29.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.852+0000 7f65ce7fc640 10 client.?.objecter ms_handle_connect 0x7f65d80a98f0 2024-03-25T19:44:29.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.852+0000 7f65ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:29.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.859+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256839679 0 0) 0x7f65d8139720 con 0x7f65d81430c0 2024-03-25T19:44:29.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.859+0000 7f65ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:29.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.859+0000 7f65ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:29.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.859+0000 7f65ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:29.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.859+0000 7f65ce7fc640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:44:29.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.859+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc004eb0 con 0x7f65d81430c0 2024-03-25T19:44:29.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.863+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578983431 0 0) 0x7f65d8138af0 con 0x7f65d80a98f0 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.863+0000 7f65ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.863+0000 7f65ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.863+0000 7f65ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.863+0000 7f65ce7fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.863+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d8139720 con 0x7f65d80a98f0 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.864+0000 7f65ce7fc640 10 client.?.objecter ms_handle_connect 0x7f65d813f710 2024-03-25T19:44:29.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.864+0000 7f65ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 235983160 0 0) 0x7f65bc004eb0 con 0x7f65d81430c0 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65d8138af0 con 0x7f65d81430c0 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f65d0001b30 con 0x7f65d81430c0 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:29.885 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.884+0000 7f65ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.885+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461180702 0 0) 0x7f65d8138d60 con 0x7f65d813f710 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.885+0000 7f65ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:29.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.885+0000 7f65ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:29.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.885+0000 7f65ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:29.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.885+0000 7f65ce7fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:44:29.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.885+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc004eb0 con 0x7f65d813f710 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1594907745 0 0) 0x7f65d8138af0 con 0x7f65d81430c0 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 >> v1:172.21.15.47:6790/0 conn(0x7f65d813f710 legacy=0x7f65d81372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 >> v1:172.21.15.142:6789/0 conn(0x7f65d80a98f0 legacy=0x7f65d8136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 1 monclient: found mon.a 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:29.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d81379e0 con 0x7f65d81430c0 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 5 monclient: authenticate success, global_id 4668 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 10 monclient: _renew_subs 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d81da8f0 con 0x7f65d81430c0 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 10 monclient: _renew_subs 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d813c380 con 0x7f65d81430c0 2024-03-25T19:44:29.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.903+0000 7f65dfc5c640 1 librados: init done 2024-03-25T19:44:29.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.904+0000 7f65dfc5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:29.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.904+0000 7f65dfc5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:29.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.904+0000 7f65dfc5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:29.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:29.904+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d8136e50 con 0x7f65d81430c0 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65d0001fe0 con 0x7f65d81430c0 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f65d0004b60 con 0x7f65d81430c0 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65ce7fc640 10 monclient: dump: 2024-03-25T19:44:30.241 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:30.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.240+0000 7f65cffff640 4 set_mon_vals no callback set 2024-03-25T19:44:30.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.241+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f65d0004d40 con 0x7f65d81430c0 2024-03-25T19:44:30.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.242+0000 7f65dd9f8640 10 monclient: get_auth_request con 0x7f65bc06de00 auth_method 0 2024-03-25T19:44:30.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.242+0000 7f65ce7fc640 10 client.4668.objecter ms_handle_connect 0x7f65bc06de00 2024-03-25T19:44:30.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7f65d0054950 con 0x7f65d81430c0 2024-03-25T19:44:30.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 10 client.4668.objecter ms_dispatch 0x7f65d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:30.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 3 client.4668.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:30.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 3 client.4668.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:30.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 20 client.4668.objecter dump_active .. 0 homeless 2024-03-25T19:44:30.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65d0050290 con 0x7f65d81430c0 2024-03-25T19:44:30.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:30.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.249+0000 7f65ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:30.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.501+0000 7f65dfc5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:30.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.501+0000 7f65dfc5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:30.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.501+0000 7f65dfc5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:30.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.501+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65d80627b0 con 0x7f65d81430c0 2024-03-25T19:44:30.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.534+0000 7f65ce7fc640 1 -- 172.21.15.47:0/4188570079 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7f65d0052390 con 0x7f65d81430c0 2024-03-25T19:44:30.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.534+0000 7f65ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:30.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.534+0000 7f65ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:30.534 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:30.535 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 10 librados: watch_flush enter 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 10 librados: watch_flush exit 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 20 client.4668.objecter shutdown clearing up homeless session... 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 10 client.4668.objecter successfully canceled tick 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 >> v1:172.21.15.142:6800/1025302268 conn(0x7f65bc06de00 legacy=0x7f65bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 10 monclient: shutdown 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 >> v1:172.21.15.47:6789/0 conn(0x7f65d81430c0 legacy=0x7f65d80b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 shutdown_connections 2024-03-25T19:44:30.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 >> 172.21.15.47:0/4188570079 conn(0x7f65d80b7ea0 msgr2=0x7f65d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:30.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 shutdown_connections 2024-03-25T19:44:30.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 -- 172.21.15.47:0/4188570079 wait complete. 2024-03-25T19:44:30.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:30.536+0000 7f65dfc5c640 1 librados: shutdown 2024-03-25T19:44:30.546 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:37.353 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:37.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.469+0000 7fc0d67fe640 10 monclient: get_monmap_and_config 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.469+0000 7fc0d67fe640 10 monclient: build_initial_monmap 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.469+0000 7fc0d67fe640 10 monclient: monmap: 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:37.470086+0000 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:37.470086+0000 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:37.469 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.470+0000 7fc0d67fe640 1 Processor -- start 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.470+0000 7fc0d67fe640 1 -- start start 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.470+0000 7fc0d67fe640 10 monclient: init 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc0d00b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc0d00b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc0d014be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d00af940 con 0x7fc0d014be90 2024-03-25T19:44:37.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d00b0a70 con 0x7fc0d00b4d60 2024-03-25T19:44:37.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d00b0d50 con 0x7fc0d00b1730 2024-03-25T19:44:37.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:37.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.471+0000 7fc0d67fe640 10 monclient(hunting): authenticate will time out at 6192.757331s 2024-03-25T19:44:37.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.479+0000 7fc0cffff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc0d00b4d60 0x7fc0d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:36102/0 (socket says 172.21.15.47:36102) 2024-03-25T19:44:37.478 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.479+0000 7fc0cffff640 1 -- 172.21.15.47:0/3460117101 learned_addr learned my addr 172.21.15.47:0/3460117101 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.495+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882132430 0 0) 0x7fc0d00b0a70 con 0x7fc0d00b4d60 2024-03-25T19:44:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.495+0000 7fc0ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.495+0000 7fc0ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.495+0000 7fc0ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.495+0000 7fc0ceffd640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:44:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.495+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b8003520 con 0x7fc0d00b4d60 2024-03-25T19:44:37.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.503+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402684147 0 0) 0x7fc0d00af940 con 0x7fc0d014be90 2024-03-25T19:44:37.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.503+0000 7fc0ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.503+0000 7fc0ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.503+0000 7fc0ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.503+0000 7fc0ceffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:44:37.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.503+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d00b0a70 con 0x7fc0d014be90 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326277947 0 0) 0x7fc0b8003520 con 0x7fc0d00b4d60 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 >> v1:172.21.15.47:6790/0 conn(0x7fc0d00b1730 legacy=0x7fc0d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 >> v1:172.21.15.47:6789/0 conn(0x7fc0d014be90 legacy=0x7fc0d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 1 monclient: found mon.noname-c 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d00b1030 con 0x7fc0d00b4d60 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc0c0001690 con 0x7fc0d00b4d60 2024-03-25T19:44:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0ceffd640 10 monclient: dump: 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:37.511 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:37.512 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:37.512 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:37.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0d67fe640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:44:37.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.510+0000 7fc0d67fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:37.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0c00017f0 con 0x7fc0d00b4d60 2024-03-25T19:44:37.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0ceffd640 1 -- 172.21.15.47:0/3460117101 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc0c0001c00 con 0x7fc0d00b4d60 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0d67fe640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0d67fe640 4 set_mon_vals no callback set 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.551+0000 7fc0ceffd640 10 monclient: dump: 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:37.551 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:37.552 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:37.552 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:37.552 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:37.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 10 monclient: shutdown 2024-03-25T19:44:37.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:37.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 1 -- 172.21.15.47:0/3460117101 >> v1:172.21.15.142:6789/0 conn(0x7fc0d00b4d60 legacy=0x7fc0d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 1 -- 172.21.15.47:0/3460117101 shutdown_connections 2024-03-25T19:44:37.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 1 -- 172.21.15.47:0/3460117101 >> 172.21.15.47:0/3460117101 conn(0x7fc0d00617a0 msgr2=0x7fc0d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:37.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 1 -- 172.21.15.47:0/3460117101 shutdown_connections 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.552+0000 7fc0d67fe640 1 -- 172.21.15.47:0/3460117101 wait complete. 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.553+0000 7fc0d67fe640 10 monclient: build_initial_monmap 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.553+0000 7fc0d67fe640 10 monclient: monmap: 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.553+0000 7fc0d67fe640 1 librados: starting msgr at 2024-03-25T19:44:37.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.553+0000 7fc0d67fe640 1 librados: starting objecter 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.553+0000 7fc0d67fe640 1 Processor -- start 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 1 -- start start 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 1 librados: setting wanted keys 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 1 librados: calling monclient init 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 10 monclient: init 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc0d00b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:37.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc0d00b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:37.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc0d014be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:37.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d01354a0 con 0x7fc0d014be90 2024-03-25T19:44:37.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.554+0000 7fc0d67fe640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d0141ce0 con 0x7fc0d00b4d60 2024-03-25T19:44:37.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.555+0000 7fc0d67fe640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d0141fc0 con 0x7fc0d00b1730 2024-03-25T19:44:37.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.555+0000 7fc0d67fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:37.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.555+0000 7fc0d67fe640 10 monclient(hunting): authenticate will time out at 6192.840958s 2024-03-25T19:44:37.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.575+0000 7fc0cffff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc0d00b1730 0x7fc0d0134780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:57678/0 (socket says 172.21.15.47:57678) 2024-03-25T19:44:37.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.575+0000 7fc0cffff640 1 -- 172.21.15.47:0/2776702895 learned_addr learned my addr 172.21.15.47:0/2776702895 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:37.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.576+0000 7fc0ccff9640 10 client.?.objecter ms_handle_connect 0x7fc0d00b1730 2024-03-25T19:44:37.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.576+0000 7fc0ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:37.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.578+0000 7fc0ccff9640 10 client.?.objecter ms_handle_connect 0x7fc0d00b4d60 2024-03-25T19:44:37.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.578+0000 7fc0ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:37.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.580+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304237740 0 0) 0x7fc0d0141fc0 con 0x7fc0d00b1730 2024-03-25T19:44:37.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.580+0000 7fc0ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.580+0000 7fc0ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.580+0000 7fc0ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.580+0000 7fc0ccff9640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:44:37.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.580+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc004eb0 con 0x7fc0d00b1730 2024-03-25T19:44:37.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.585+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769116523 0 0) 0x7fc0d0141ce0 con 0x7fc0d00b4d60 2024-03-25T19:44:37.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.585+0000 7fc0ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:37.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.585+0000 7fc0ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:37.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.585+0000 7fc0ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:37.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.585+0000 7fc0ccff9640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:44:37.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.585+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d0141fc0 con 0x7fc0d00b4d60 2024-03-25T19:44:37.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227228909 0 0) 0x7fc0bc004eb0 con 0x7fc0d00b1730 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0d0141ce0 con 0x7fc0d00b1730 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc0c0001bc0 con 0x7fc0d00b1730 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 10 monclient(hunting): dump: 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:37.589 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:37.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.589+0000 7fc0ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:37.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.590+0000 7fc0ccff9640 10 client.?.objecter ms_handle_connect 0x7fc0d014be90 2024-03-25T19:44:37.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.590+0000 7fc0ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:37.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1649835999 0 0) 0x7fc0d0141ce0 con 0x7fc0d00b1730 2024-03-25T19:44:37.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 >> v1:172.21.15.142:6789/0 conn(0x7fc0d00b4d60 legacy=0x7fc0d01df650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 >> v1:172.21.15.47:6789/0 conn(0x7fc0d014be90 legacy=0x7fc0d01dfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 1 monclient: found mon.c 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d01356f0 con 0x7fc0d00b1730 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 5 monclient: authenticate success, global_id 4664 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 10 monclient: _renew_subs 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:44:37.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d0140380 con 0x7fc0d00b1730 2024-03-25T19:44:37.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:37.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 10 monclient: _renew_subs 2024-03-25T19:44:37.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:44:37.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d013b860 con 0x7fc0d00b1730 2024-03-25T19:44:37.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.594+0000 7fc0d67fe640 1 librados: init done 2024-03-25T19:44:37.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.596+0000 7fc0d67fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:37.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.596+0000 7fc0d67fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:37.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.596+0000 7fc0d67fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:44:37.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.596+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d00627b0 con 0x7fc0d00b1730 2024-03-25T19:44:37.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0c00011f0 con 0x7fc0d00b1730 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc0c0004cb0 con 0x7fc0d00b1730 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 10 monclient: dump: 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:37.920 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:37.921 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:37.921 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:37.921 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:37.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ce7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:37.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.920+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fc0c001ad00 con 0x7fc0d00b1730 2024-03-25T19:44:37.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.921+0000 7fc0cf7fe640 10 monclient: get_auth_request con 0x7fc0bc06dcc0 auth_method 0 2024-03-25T19:44:37.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.921+0000 7fc0ccff9640 10 client.4664.objecter ms_handle_connect 0x7fc0bc06dcc0 2024-03-25T19:44:37.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3393621215 0 0) 0x7fc0c0054970 con 0x7fc0d00b1730 2024-03-25T19:44:37.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 10 client.4664.objecter ms_dispatch 0x7fc0d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:37.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 3 client.4664.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:37.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 3 client.4664.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:37.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-25T19:44:37.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0c00502b0 con 0x7fc0d00b1730 2024-03-25T19:44:37.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:37.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:37.928+0000 7fc0ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:38.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.176+0000 7fc0d67fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:38.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.176+0000 7fc0d67fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:38.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.176+0000 7fc0d67fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:44:38.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.176+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d0137cc0 con 0x7fc0d00b1730 2024-03-25T19:44:38.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.204+0000 7fc0ccff9640 1 -- 172.21.15.47:0/2776702895 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2438202575) 0x7fc0c00523b0 con 0x7fc0d00b1730 2024-03-25T19:44:38.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.204+0000 7fc0ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:38.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.204+0000 7fc0ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:38.204 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:38.204 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:34:45.871047+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:34:45.871047+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 10 librados: watch_flush enter 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 10 librados: watch_flush exit 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 10 client.4664.objecter successfully canceled tick 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 >> v1:172.21.15.142:6800/1025302268 conn(0x7fc0bc06dcc0 legacy=0x7fc0bc070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 10 monclient: shutdown 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 >> v1:172.21.15.47:6790/0 conn(0x7fc0d00b1730 legacy=0x7fc0d0134780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 shutdown_connections 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 >> 172.21.15.47:0/2776702895 conn(0x7fc0d00617a0 msgr2=0x7fc0d00b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:38.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 shutdown_connections 2024-03-25T19:44:38.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 -- 172.21.15.47:0/2776702895 wait complete. 2024-03-25T19:44:38.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:38.207+0000 7fc0d67fe640 1 librados: shutdown 2024-03-25T19:44:38.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:44.144 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:21.379 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.150 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:21.340 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.172 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:21.407 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.193 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:21.383 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.195 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:21.431 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.206 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:21.396 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.228 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:21.464 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.229 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:21.465 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:44.297 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:21.532 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.027 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:45.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.144+0000 7fa3b18e4640 10 monclient: get_monmap_and_config 2024-03-25T19:44:45.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.144+0000 7fa3b18e4640 10 monclient: build_initial_monmap 2024-03-25T19:44:45.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.144+0000 7fa3b18e4640 10 monclient: monmap: 2024-03-25T19:44:45.144 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:45.145735+0000 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:45.145735+0000 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:45.145 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:45.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.145+0000 7fa3b18e4640 1 Processor -- start 2024-03-25T19:44:45.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.145+0000 7fa3b18e4640 1 -- start start 2024-03-25T19:44:45.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.145+0000 7fa3b18e4640 10 monclient: init 2024-03-25T19:44:45.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:45.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac142520 con 0x7fa3ac13f710 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0b28a0 con 0x7fa3ac1430c0 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0b2a30 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:45.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.146+0000 7fa3b18e4640 10 monclient(hunting): authenticate will time out at 6200.432851s 2024-03-25T19:44:45.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.153+0000 7fa3aaffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa3ac0a98f0 0x7fa3ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33854/0 (socket says 172.21.15.47:33854) 2024-03-25T19:44:45.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.153+0000 7fa3aaffd640 1 -- 172.21.15.47:0/1276215488 learned_addr learned my addr 172.21.15.47:0/1276215488 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:45.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.159+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043587280 0 0) 0x7fa3ac0b2a30 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.159+0000 7fa3a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.159+0000 7fa3a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.159+0000 7fa3a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.159+0000 7fa3a9ffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:44:45.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.159+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa394003520 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.177+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269270885 0 0) 0x7fa3ac142520 con 0x7fa3ac13f710 2024-03-25T19:44:45.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.177+0000 7fa3a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.177+0000 7fa3a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.177+0000 7fa3a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.177+0000 7fa3a9ffb640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:44:45.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.177+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac0b2a30 con 0x7fa3ac13f710 2024-03-25T19:44:45.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2785130616 0 0) 0x7fa394003520 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 >> v1:172.21.15.142:6789/0 conn(0x7fa3ac1430c0 legacy=0x7fa3ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 >> v1:172.21.15.47:6789/0 conn(0x7fa3ac13f710 legacy=0x7fa3ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 1 monclient: found mon.noname-b 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0b44c0 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa39c001690 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3b18e4640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:44:45.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3b18e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3a9ffb640 10 monclient: dump: 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:45.186 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:45.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.183+0000 7fa3b18e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa39c0017f0 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3a9ffb640 1 -- 172.21.15.47:0/1276215488 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa39c001c00 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3b18e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3b18e4640 4 set_mon_vals no callback set 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.190+0000 7fa3a9ffb640 10 monclient: dump: 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.190 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.191 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.191 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:45.191 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:45.191 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:45.191 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:45.191 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:45.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 10 monclient: shutdown 2024-03-25T19:44:45.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1276215488 >> v1:172.21.15.47:6790/0 conn(0x7fa3ac0a98f0 legacy=0x7fa3ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1276215488 shutdown_connections 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1276215488 >> 172.21.15.47:0/1276215488 conn(0x7fa3ac0b7ea0 msgr2=0x7fa3ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1276215488 shutdown_connections 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.191+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1276215488 wait complete. 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.192+0000 7fa3b18e4640 10 monclient: build_initial_monmap 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.192+0000 7fa3b18e4640 10 monclient: monmap: 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:45.194 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.192+0000 7fa3b18e4640 1 librados: starting msgr at 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.192+0000 7fa3b18e4640 1 librados: starting objecter 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.192+0000 7fa3b18e4640 1 Processor -- start 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 1 -- start start 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 1 librados: setting wanted keys 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 1 librados: calling monclient init 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient: init 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac0a98f0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:45.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac13f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac1430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0abd30 con 0x7fa3ac13f710 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0abfa0 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac1d77a0 con 0x7fa3ac1430c0 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.193+0000 7fa3b18e4640 10 monclient(hunting): authenticate will time out at 6200.479879s 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.194+0000 7fa3aaffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa3ac0a98f0 0x7fa3ac1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:47112/0 (socket says 172.21.15.47:47112) 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.194+0000 7fa3aaffd640 1 -- 172.21.15.47:0/1575642251 learned_addr learned my addr 172.21.15.47:0/1575642251 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.195+0000 7fa393fff640 10 client.?.objecter ms_handle_connect 0x7fa3ac0a98f0 2024-03-25T19:44:45.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.195+0000 7fa393fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:45.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.200+0000 7fa393fff640 10 client.?.objecter ms_handle_connect 0x7fa3ac1430c0 2024-03-25T19:44:45.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.200+0000 7fa393fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:45.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.216+0000 7fa393fff640 10 client.?.objecter ms_handle_connect 0x7fa3ac13f710 2024-03-25T19:44:45.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.216+0000 7fa393fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.278+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636685454 0 0) 0x7fa3ac0abd30 con 0x7fa3ac13f710 2024-03-25T19:44:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.278+0000 7fa393fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.278+0000 7fa393fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.278+0000 7fa393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.278+0000 7fa393fff640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:44:45.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.278+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa398004eb0 con 0x7fa3ac13f710 2024-03-25T19:44:45.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.300+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057769610 0 0) 0x7fa3ac1d77a0 con 0x7fa3ac1430c0 2024-03-25T19:44:45.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.300+0000 7fa393fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.300+0000 7fa393fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.300+0000 7fa393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.300+0000 7fa393fff640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:44:45.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.300+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac0abd30 con 0x7fa3ac1430c0 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1789456128 0 0) 0x7fa398004eb0 con 0x7fa3ac13f710 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ac1d77a0 con 0x7fa3ac13f710 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa3a0001930 con 0x7fa3ac13f710 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 10 monclient(hunting): dump: 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.304 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:45.305 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:45.305 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:45.305 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:45.305 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:45.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.304+0000 7fa393fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:45.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.311+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448427205 0 0) 0x7fa3ac0abfa0 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.311+0000 7fa393fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:45.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.311+0000 7fa393fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:45.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.311+0000 7fa393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:45.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.311+0000 7fa393fff640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:44:45.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.311+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa398004eb0 con 0x7fa3ac0a98f0 2024-03-25T19:44:45.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 176076469 0 0) 0x7fa3ac1d77a0 con 0x7fa3ac13f710 2024-03-25T19:44:45.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 >> v1:172.21.15.47:6790/0 conn(0x7fa3ac1430c0 legacy=0x7fa3ac1385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 >> v1:172.21.15.142:6789/0 conn(0x7fa3ac0a98f0 legacy=0x7fa3ac1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 1 monclient: found mon.a 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.319+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac1d83d0 con 0x7fa3ac13f710 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa393fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa393fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 5 monclient: authenticate success, global_id 4680 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 monclient: _renew_subs 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac1d41e0 con 0x7fa3ac13f710 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:45.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 monclient: _renew_subs 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac1d5110 con 0x7fa3ac13f710 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 1 librados: init done 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:45.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.320+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac1d7930 con 0x7fa3ac13f710 2024-03-25T19:44:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3a0003610 con 0x7fa3ac13f710 2024-03-25T19:44:45.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fa3a0003a20 con 0x7fa3ac13f710 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 10 monclient: dump: 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:45.652 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:45.653 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:45.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa3a97fa640 4 set_mon_vals no callback set 2024-03-25T19:44:45.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.652+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fa3a0003c40 con 0x7fa3ac13f710 2024-03-25T19:44:45.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.653+0000 7fa3aaffd640 10 monclient: get_auth_request con 0x7fa39806de00 auth_method 0 2024-03-25T19:44:45.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.653+0000 7fa393fff640 10 client.4680.objecter ms_handle_connect 0x7fa39806de00 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1684707788 0 0) 0x7fa3a0053600 con 0x7fa3ac13f710 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 10 client.4680.objecter ms_dispatch 0x7fa3ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 3 client.4680.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 3 client.4680.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 20 client.4680.objecter dump_active .. 0 homeless 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3a004f110 con 0x7fa3ac13f710 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:45.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.658+0000 7fa393fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:45.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.898+0000 7fa3b18e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:45.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.898+0000 7fa3b18e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:45.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.898+0000 7fa3b18e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:45.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.898+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3ac136a20 con 0x7fa3ac13f710 2024-03-25T19:44:45.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.927+0000 7fa393fff640 1 -- 172.21.15.47:0/1575642251 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7086 (unknown 238102980 0 214449246) 0x7fa3a0051040 con 0x7fa3ac13f710 2024-03-25T19:44:45.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.927+0000 7fa393fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:45.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.927+0000 7fa393fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:45.927 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:45.927 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:43.896873+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:43.896873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 10 librados: watch_flush enter 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 10 librados: watch_flush exit 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 20 client.4680.objecter shutdown clearing up homeless session... 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 10 client.4680.objecter successfully canceled tick 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 >> v1:172.21.15.142:6800/1025302268 conn(0x7fa39806de00 legacy=0x7fa3980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 10 monclient: shutdown 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:45.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 >> v1:172.21.15.47:6789/0 conn(0x7fa3ac13f710 legacy=0x7fa3ac1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:45.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 shutdown_connections 2024-03-25T19:44:45.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 >> 172.21.15.47:0/1575642251 conn(0x7fa3ac0b7ea0 msgr2=0x7fa3ac13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:45.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 shutdown_connections 2024-03-25T19:44:45.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 -- 172.21.15.47:0/1575642251 wait complete. 2024-03-25T19:44:45.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:45.929+0000 7fa3b18e4640 1 librados: shutdown 2024-03-25T19:44:45.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:52.749 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:52.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.863+0000 7f12fd845640 10 monclient: get_monmap_and_config 2024-03-25T19:44:52.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.863+0000 7f12fd845640 10 monclient: build_initial_monmap 2024-03-25T19:44:52.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.863+0000 7f12fd845640 10 monclient: monmap: 2024-03-25T19:44:52.863 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:44:52.864723+0000 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:44:52.864723+0000 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:44:52.864 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.864+0000 7f12fd845640 1 Processor -- start 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.864+0000 7f12fd845640 1 -- start start 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.864+0000 7f12fd845640 10 monclient: init 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient(hunting): picked mon.noname-b con 0x7f12f80a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:52.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient(hunting): picked mon.noname-a con 0x7f12f813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:52.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient(hunting): picked mon.noname-c con 0x7f12f81430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:52.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f8142520 con 0x7f12f813f710 2024-03-25T19:44:52.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f80b28a0 con 0x7f12f81430c0 2024-03-25T19:44:52.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f80b2a30 con 0x7f12f80a98f0 2024-03-25T19:44:52.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:52.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.865+0000 7f12fd845640 10 monclient(hunting): authenticate will time out at 6208.151621s 2024-03-25T19:44:52.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.880+0000 7f12effff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f12f813f710 0x7f12f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42984/0 (socket says 172.21.15.47:42984) 2024-03-25T19:44:52.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.880+0000 7f12effff640 1 -- 172.21.15.47:0/962752686 learned_addr learned my addr 172.21.15.47:0/962752686 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:52.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.895+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409061872 0 0) 0x7f12f80b2a30 con 0x7f12f80a98f0 2024-03-25T19:44:52.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.895+0000 7f12f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:52.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.895+0000 7f12f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:52.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.895+0000 7f12f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:52.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.895+0000 7f12f67fc640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:44:52.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.895+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e4003520 con 0x7f12f80a98f0 2024-03-25T19:44:52.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.907+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394766254 0 0) 0x7f12f8142520 con 0x7f12f813f710 2024-03-25T19:44:52.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.907+0000 7f12f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:52.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.907+0000 7f12f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:52.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.907+0000 7f12f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:52.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.907+0000 7f12f67fc640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:44:52.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.907+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f80b2a30 con 0x7f12f813f710 2024-03-25T19:44:52.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.910+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879067207 0 0) 0x7f12f80b28a0 con 0x7f12f81430c0 2024-03-25T19:44:52.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.910+0000 7f12f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:52.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.910+0000 7f12f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:52.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.910+0000 7f12f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:52.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.910+0000 7f12f67fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:44:52.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.910+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f8142520 con 0x7f12f81430c0 2024-03-25T19:44:52.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2491667228 0 0) 0x7f12f8142520 con 0x7f12f81430c0 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 >> v1:172.21.15.47:6790/0 conn(0x7f12f80a98f0 legacy=0x7f12f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 >> v1:172.21.15.47:6789/0 conn(0x7f12f813f710 legacy=0x7f12f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 1 monclient: found mon.noname-c 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f80b44c0 con 0x7f12f81430c0 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f12f0001930 con 0x7f12f81430c0 2024-03-25T19:44:52.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12f67fc640 10 monclient: dump: 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12fd845640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:44:52.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.921+0000 7f12fd845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:52.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12f0001a90 con 0x7f12f81430c0 2024-03-25T19:44:52.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:52.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12f67fc640 1 -- 172.21.15.47:0/962752686 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f12f0001ea0 con 0x7f12f81430c0 2024-03-25T19:44:52.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12f67fc640 10 monclient: dump: 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:52.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 4 set_mon_vals no callback set 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 10 monclient: shutdown 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 1 -- 172.21.15.47:0/962752686 >> v1:172.21.15.142:6789/0 conn(0x7f12f81430c0 legacy=0x7f12f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 1 -- 172.21.15.47:0/962752686 shutdown_connections 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.959+0000 7f12fd845640 1 -- 172.21.15.47:0/962752686 >> 172.21.15.47:0/962752686 conn(0x7f12f80b7ea0 msgr2=0x7f12f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.960+0000 7f12fd845640 1 -- 172.21.15.47:0/962752686 shutdown_connections 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.960+0000 7f12fd845640 1 -- 172.21.15.47:0/962752686 wait complete. 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.960+0000 7f12fd845640 10 monclient: build_initial_monmap 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.960+0000 7f12fd845640 10 monclient: monmap: 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:52.962 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 1 librados: starting msgr at 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 1 librados: starting objecter 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 1 Processor -- start 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 1 -- start start 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 1 librados: setting wanted keys 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 1 librados: calling monclient init 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 10 monclient: init 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:52.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 10 monclient(hunting): picked mon.noname-a con 0x7f12f80a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 10 monclient(hunting): picked mon.noname-b con 0x7f12f813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.961+0000 7f12fd845640 10 monclient(hunting): picked mon.noname-c con 0x7f12f81430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.962+0000 7f12fd845640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f80abe50 con 0x7f12f80a98f0 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.962+0000 7f12fd845640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f8138f20 con 0x7f12f813f710 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.962+0000 7f12fd845640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f81d51f0 con 0x7f12f81430c0 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.962+0000 7f12fd845640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:52.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.962+0000 7f12fd845640 10 monclient(hunting): authenticate will time out at 6208.248065s 2024-03-25T19:44:52.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.970+0000 7f12effff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f12f813f710 0x7f12f81d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:47218/0 (socket says 172.21.15.47:47218) 2024-03-25T19:44:52.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.970+0000 7f12effff640 1 -- 172.21.15.47:0/1977164896 learned_addr learned my addr 172.21.15.47:0/1977164896 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:44:52.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.974+0000 7f12f4ff9640 10 client.?.objecter ms_handle_connect 0x7f12f813f710 2024-03-25T19:44:52.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.974+0000 7f12f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:52.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.974+0000 7f12f4ff9640 10 client.?.objecter ms_handle_connect 0x7f12f80a98f0 2024-03-25T19:44:52.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.974+0000 7f12f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:52.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.983+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184236947 0 0) 0x7f12f80abe50 con 0x7f12f80a98f0 2024-03-25T19:44:52.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.983+0000 7f12f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:52.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.983+0000 7f12f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:52.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.983+0000 7f12f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:52.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.983+0000 7f12f4ff9640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:44:52.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.983+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d8004eb0 con 0x7f12f80a98f0 2024-03-25T19:44:52.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.989+0000 7f12f4ff9640 10 client.?.objecter ms_handle_connect 0x7f12f81430c0 2024-03-25T19:44:52.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.989+0000 7f12f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:52.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.991+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275253626 0 0) 0x7f12f8138f20 con 0x7f12f813f710 2024-03-25T19:44:52.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.991+0000 7f12f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:52.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.991+0000 7f12f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:52.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.991+0000 7f12f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:52.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.991+0000 7f12f4ff9640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:44:52.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:52.991+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f80abe50 con 0x7f12f813f710 2024-03-25T19:44:53.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.008+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161636000 0 0) 0x7f12f81d51f0 con 0x7f12f81430c0 2024-03-25T19:44:53.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.008+0000 7f12f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.008+0000 7f12f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.008+0000 7f12f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.008+0000 7f12f4ff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:44:53.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.008+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f8138f20 con 0x7f12f81430c0 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.010+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264420182 0 0) 0x7f12d8004eb0 con 0x7f12f80a98f0 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.011+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12f81d51f0 con 0x7f12f80a98f0 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.011+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f12e8001950 con 0x7f12f80a98f0 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.011+0000 7f12f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.011+0000 7f12f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.011+0000 7f12f4ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:53.010 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:53.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.011+0000 7f12f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.019+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323579402 0 0) 0x7f12f80abe50 con 0x7f12f813f710 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.019+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12d8004eb0 con 0x7f12f813f710 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.019+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f12e00019f0 con 0x7f12f813f710 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.019+0000 7f12f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.019+0000 7f12f4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.019+0000 7f12f4ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:53.019 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:53.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2816750022 0 0) 0x7f12f81d51f0 con 0x7f12f80a98f0 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 >> v1:172.21.15.47:6790/0 conn(0x7f12f81430c0 legacy=0x7f12f81e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 >> v1:172.21.15.142:6789/0 conn(0x7f12f813f710 legacy=0x7f12f81d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 1 monclient: found mon.a 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f81d7730 con 0x7f12f80a98f0 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 10 monclient: _renew_subs 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:53.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f81d8360 con 0x7f12f80a98f0 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 10 monclient: _renew_subs 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f8137ce0 con 0x7f12f80a98f0 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.029+0000 7f12fd845640 1 librados: init done 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.030+0000 7f12fd845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.030+0000 7f12fd845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.030+0000 7f12fd845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:53.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.030+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f81d7590 con 0x7f12f80a98f0 2024-03-25T19:44:53.359 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12e80035f0 con 0x7f12f80a98f0 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f12e8003760 con 0x7f12f80a98f0 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f4ff9640 10 monclient: dump: 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:44:53.360 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:44:53.361 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:44:53.361 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:44:53.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.360+0000 7f12f5ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:53.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.361+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f12e8019730 con 0x7f12f80a98f0 2024-03-25T19:44:53.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.361+0000 7f12effff640 10 monclient: get_auth_request con 0x7f12d806df70 auth_method 0 2024-03-25T19:44:53.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.362+0000 7f12f4ff9640 10 client.4686.objecter ms_handle_connect 0x7f12d806df70 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f12e8053370 con 0x7f12f80a98f0 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 10 client.4686.objecter ms_dispatch 0x7f12f8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 3 client.4686.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 3 client.4686.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12e804ee80 con 0x7f12f80a98f0 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:53.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.367+0000 7f12f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:53.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.620+0000 7f12fd845640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:53.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.620+0000 7f12fd845640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:53.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.620+0000 7f12fd845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:44:53.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.620+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f81d7970 con 0x7f12f80a98f0 2024-03-25T19:44:53.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.680+0000 7f12f4ff9640 1 -- 172.21.15.47:0/1977164896 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f12f81d7970 con 0x7f12f80a98f0 2024-03-25T19:44:53.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.680+0000 7f12f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:53.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.680+0000 7f12f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:53.680 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:44:53.680 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 10 librados: watch_flush enter 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 10 librados: watch_flush exit 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 10 client.4686.objecter successfully canceled tick 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 >> v1:172.21.15.142:6800/1025302268 conn(0x7f12d806df70 legacy=0x7f12d8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 10 monclient: shutdown 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 >> v1:172.21.15.47:6789/0 conn(0x7f12f80a98f0 legacy=0x7f12f8138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 shutdown_connections 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 >> 172.21.15.47:0/1977164896 conn(0x7f12f80b7ea0 msgr2=0x7f12f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 shutdown_connections 2024-03-25T19:44:53.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.682+0000 7f12fd845640 1 -- 172.21.15.47:0/1977164896 wait complete. 2024-03-25T19:44:53.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:44:53.683+0000 7f12fd845640 1 librados: shutdown 2024-03-25T19:44:53.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:00.500 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.596+0000 7f854f77c640 10 monclient: get_monmap_and_config 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.596+0000 7f854f77c640 10 monclient: build_initial_monmap 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.596+0000 7f854f77c640 10 monclient: monmap: 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:00.597447+0000 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:00.597447+0000 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:00.596 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:00.597 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:00.597 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:00.597 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.597+0000 7f854f77c640 1 Processor -- start 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.597+0000 7f854f77c640 1 -- start start 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.597+0000 7f854f77c640 10 monclient: init 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient(hunting): picked mon.noname-a con 0x7f85480b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:00.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient(hunting): picked mon.noname-b con 0x7f85480b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:00.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient(hunting): picked mon.noname-c con 0x7f854814be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:00.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480af940 con 0x7f85480b4d60 2024-03-25T19:45:00.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480b0a70 con 0x7f854814be90 2024-03-25T19:45:00.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480b0d50 con 0x7f85480b1730 2024-03-25T19:45:00.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:00.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.598+0000 7f854f77c640 10 monclient(hunting): authenticate will time out at 6215.884637s 2024-03-25T19:45:00.607 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.608+0000 7f854d518640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f85480b4d60 0x7f854813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50682/0 (socket says 172.21.15.47:50682) 2024-03-25T19:45:00.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.608+0000 7f854d518640 1 -- 172.21.15.47:0/1754557833 learned_addr learned my addr 172.21.15.47:0/1754557833 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:00.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.624+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478599018 0 0) 0x7f85480af940 con 0x7f85480b4d60 2024-03-25T19:45:00.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.624+0000 7f853ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.624+0000 7f853ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.624+0000 7f853ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.624+0000 7f853ffff640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:45:00.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.625+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534003520 con 0x7f85480b4d60 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3105687631 0 0) 0x7f8534003520 con 0x7f85480b4d60 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 >> v1:172.21.15.47:6790/0 conn(0x7f85480b1730 legacy=0x7f85480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 >> v1:172.21.15.142:6789/0 conn(0x7f854814be90 legacy=0x7f854814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 1 monclient: found mon.noname-a 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:45:00.639 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85480b1030 con 0x7f85480b4d60 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: _finish_auth 0 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8544001690 con 0x7f85480b4d60 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f853ffff640 10 monclient: dump: 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:00.640 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f854f77c640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:45:00.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.639+0000 7f854f77c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:00.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85440017f0 con 0x7f85480b4d60 2024-03-25T19:45:00.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f853ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:00.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f853ffff640 1 -- 172.21.15.47:0/1754557833 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8544001c00 con 0x7f85480b4d60 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f853ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f853ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f853ffff640 10 monclient: dump: 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f854f77c640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:00.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f854f77c640 4 set_mon_vals no callback set 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f854f77c640 10 monclient: shutdown 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f854f77c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.737+0000 7f854f77c640 1 -- 172.21.15.47:0/1754557833 >> v1:172.21.15.47:6789/0 conn(0x7f85480b4d60 legacy=0x7f854813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.738+0000 7f854f77c640 1 -- 172.21.15.47:0/1754557833 shutdown_connections 2024-03-25T19:45:00.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.738+0000 7f854f77c640 1 -- 172.21.15.47:0/1754557833 >> 172.21.15.47:0/1754557833 conn(0x7f85480617a0 msgr2=0x7f8548061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.738+0000 7f854f77c640 1 -- 172.21.15.47:0/1754557833 shutdown_connections 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.738+0000 7f854f77c640 1 -- 172.21.15.47:0/1754557833 wait complete. 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 10 monclient: build_initial_monmap 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 10 monclient: monmap: 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 1 librados: starting msgr at 2024-03-25T19:45:00.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 1 librados: starting objecter 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 1 Processor -- start 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 1 -- start start 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 1 librados: setting wanted keys 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 1 librados: calling monclient init 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.739+0000 7f854f77c640 10 monclient: init 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient(hunting): picked mon.noname-c con 0x7f85480b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient(hunting): picked mon.noname-a con 0x7f85480b4d60 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient(hunting): picked mon.noname-b con 0x7f854814be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8548142f70 con 0x7f85480b4d60 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85481411b0 con 0x7f854814be90 2024-03-25T19:45:00.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8548143250 con 0x7f85480b1730 2024-03-25T19:45:00.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:00.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854f77c640 10 monclient(hunting): authenticate will time out at 6216.026243s 2024-03-25T19:45:00.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854dd19640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f854814be90 0x7f8548141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:39820/0 (socket says 172.21.15.47:39820) 2024-03-25T19:45:00.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.740+0000 7f854dd19640 1 -- 172.21.15.47:0/338724925 learned_addr learned my addr 172.21.15.47:0/338724925 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:00.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.741+0000 7f853dffb640 10 client.?.objecter ms_handle_connect 0x7f854814be90 2024-03-25T19:45:00.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.741+0000 7f853dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.755+0000 7f853dffb640 10 client.?.objecter ms_handle_connect 0x7f85480b4d60 2024-03-25T19:45:00.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.755+0000 7f853dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:00.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.758+0000 7f853dffb640 10 client.?.objecter ms_handle_connect 0x7f85480b1730 2024-03-25T19:45:00.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.758+0000 7f853dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.767+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478936818 0 0) 0x7f8548142f70 con 0x7f85480b4d60 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.767+0000 7f853dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.767+0000 7f853dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.767+0000 7f853dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.767+0000 7f853dffb640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.767+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852c004eb0 con 0x7f85480b4d60 2024-03-25T19:45:00.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.768+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914959974 0 0) 0x7f85481411b0 con 0x7f854814be90 2024-03-25T19:45:00.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.768+0000 7f853dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.768+0000 7f853dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.768+0000 7f853dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.768+0000 7f853dffb640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:45:00.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.768+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8548142f70 con 0x7f854814be90 2024-03-25T19:45:00.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2320530276 0 0) 0x7f8548142f70 con 0x7f854814be90 2024-03-25T19:45:00.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85481411b0 con 0x7f854814be90 2024-03-25T19:45:00.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8540001bd0 con 0x7f854814be90 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 10 monclient(hunting): dump: 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:00.776 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:00.777 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:00.777 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:00.777 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:00.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.776+0000 7f853dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:00.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.781+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481191604 0 0) 0x7f8548143250 con 0x7f85480b1730 2024-03-25T19:45:00.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.781+0000 7f853dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:00.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.781+0000 7f853dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:00.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.781+0000 7f853dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:00.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.781+0000 7f853dffb640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:45:00.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.781+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8548142f70 con 0x7f85480b1730 2024-03-25T19:45:00.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.791+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1591257368 0 0) 0x7f852c004eb0 con 0x7f85480b4d60 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.791+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8548143250 con 0x7f85480b4d60 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.791+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8538001930 con 0x7f85480b4d60 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.791+0000 7f853dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.791+0000 7f853dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.791+0000 7f853dffb640 10 monclient(hunting): dump: 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:00.791 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:00.792 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 361861843 0 0) 0x7f85481411b0 con 0x7f854814be90 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 >> v1:172.21.15.47:6790/0 conn(0x7f85480b1730 legacy=0x7f85481372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 >> v1:172.21.15.47:6789/0 conn(0x7f85480b4d60 legacy=0x7f854813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 1 monclient: found mon.b 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8548140c90 con 0x7f854814be90 2024-03-25T19:45:00.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f853dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 5 monclient: authenticate success, global_id 4687 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 10 monclient: _renew_subs 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85481379c0 con 0x7f854814be90 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 10 monclient: _renew_subs 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f854813d9e0 con 0x7f854814be90 2024-03-25T19:45:00.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.800+0000 7f854f77c640 1 librados: init done 2024-03-25T19:45:00.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.802+0000 7f854f77c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:00.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.802+0000 7f854f77c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:00.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.802+0000 7f854f77c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:00.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:00.802+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85480627b0 con 0x7f854814be90 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85400020a0 con 0x7f854814be90 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f8540003b30 con 0x7f854814be90 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853dffb640 10 monclient: dump: 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:01.082 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:01.083 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:01.083 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:01.083 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:01.083 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:01.083 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:01.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.082+0000 7f853f7fe640 4 set_mon_vals no callback set 2024-03-25T19:45:01.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.083+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f8540003d10 con 0x7f854814be90 2024-03-25T19:45:01.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.083+0000 7f854d518640 10 monclient: get_auth_request con 0x7f852c06ded0 auth_method 0 2024-03-25T19:45:01.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.084+0000 7f853dffb640 10 client.4687.objecter ms_handle_connect 0x7f852c06ded0 2024-03-25T19:45:01.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f8540054a60 con 0x7f854814be90 2024-03-25T19:45:01.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 10 client.4687.objecter ms_dispatch 0x7f8548000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:01.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 3 client.4687.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:01.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 3 client.4687.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:01.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T19:45:01.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85400503a0 con 0x7f854814be90 2024-03-25T19:45:01.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:01.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.088+0000 7f853dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:01.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.347+0000 7f854f77c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:01.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.347+0000 7f854f77c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:01.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.347+0000 7f854f77c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:01.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.347+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8548140e20 con 0x7f854814be90 2024-03-25T19:45:01.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.372+0000 7f853dffb640 1 -- 172.21.15.47:0/338724925 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f85400524a0 con 0x7f854814be90 2024-03-25T19:45:01.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.372+0000 7f853dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:01.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.372+0000 7f853dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:01.371 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:01.372 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:01.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.374+0000 7f854f77c640 10 librados: watch_flush enter 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.374+0000 7f854f77c640 10 librados: watch_flush exit 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.374+0000 7f854f77c640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.374+0000 7f854f77c640 10 client.4687.objecter successfully canceled tick 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 >> v1:172.21.15.142:6800/1025302268 conn(0x7f852c06ded0 legacy=0x7f852c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 10 monclient: shutdown 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 >> v1:172.21.15.142:6789/0 conn(0x7f854814be90 legacy=0x7f8548141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 shutdown_connections 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 >> 172.21.15.47:0/338724925 conn(0x7f85480617a0 msgr2=0x7f85480b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 shutdown_connections 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 -- 172.21.15.47:0/338724925 wait complete. 2024-03-25T19:45:01.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:01.375+0000 7f854f77c640 1 librados: shutdown 2024-03-25T19:45:01.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:04.088 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:41.323 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.095 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:41.331 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.134 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:41.369 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.140 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:41.376 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.151 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:41.387 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.165 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:41.355 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.172 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:41.362 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.181 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:41.417 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:04.196 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:10:41.386 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:08.194 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:08.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.312+0000 7f81e8383640 10 monclient: get_monmap_and_config 2024-03-25T19:45:08.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.312+0000 7f81e8383640 10 monclient: build_initial_monmap 2024-03-25T19:45:08.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.312+0000 7f81e8383640 10 monclient: monmap: 2024-03-25T19:45:08.312 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:08.312 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:08.313297+0000 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:08.313297+0000 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.313+0000 7f81e8383640 1 Processor -- start 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.313+0000 7f81e8383640 1 -- start start 2024-03-25T19:45:08.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.313+0000 7f81e8383640 10 monclient: init 2024-03-25T19:45:08.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:08.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient(hunting): picked mon.noname-c con 0x7f81e013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient(hunting): picked mon.noname-b con 0x7f81e01430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0142520 con 0x7f81e00a98f0 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b28a0 con 0x7f81e013f710 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b2a30 con 0x7f81e01430c0 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:08.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.314+0000 7f81e8383640 10 monclient(hunting): authenticate will time out at 6223.600094s 2024-03-25T19:45:08.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.323+0000 7f81e591e640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f81e013f710 0x7f81e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:41068/0 (socket says 172.21.15.47:41068) 2024-03-25T19:45:08.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.323+0000 7f81e591e640 1 -- 172.21.15.47:0/2758512983 learned_addr learned my addr 172.21.15.47:0/2758512983 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:08.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.330+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328890856 0 0) 0x7f81e00b28a0 con 0x7f81e013f710 2024-03-25T19:45:08.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.330+0000 7f81e511d640 10 monclient(hunting): _negotiate 2024-03-25T19:45:08.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.330+0000 7f81e511d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:08.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.330+0000 7f81e511d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:08.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.330+0000 7f81e511d640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:45:08.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.330+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc003520 con 0x7f81e013f710 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2599667446 0 0) 0x7f81cc003520 con 0x7f81e013f710 2024-03-25T19:45:08.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 >> v1:172.21.15.47:6790/0 conn(0x7f81e01430c0 legacy=0x7f81e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 >> v1:172.21.15.47:6789/0 conn(0x7f81e00a98f0 legacy=0x7f81e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: _finish_hunting 0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 1 monclient: found mon.noname-c 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00b44c0 con 0x7f81e013f710 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: _finish_auth 0 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: _check_auth_tickets 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f81d0001690 con 0x7f81e013f710 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:08.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e511d640 10 monclient: dump: 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:08.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e8383640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:45:08.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.337+0000 7f81e8383640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:08.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d0001860 con 0x7f81e013f710 2024-03-25T19:45:08.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e511d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e511d640 1 -- 172.21.15.47:0/2758512983 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f81d0001cb0 con 0x7f81e013f710 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e8383640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e8383640 4 set_mon_vals no callback set 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e511d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e511d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.344+0000 7f81e511d640 10 monclient: dump: 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.345 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:08.346 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:08.346 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:08.346 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:08.346 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 10 monclient: shutdown 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 1 -- 172.21.15.47:0/2758512983 >> v1:172.21.15.142:6789/0 conn(0x7f81e013f710 legacy=0x7f81e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 1 -- 172.21.15.47:0/2758512983 shutdown_connections 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 1 -- 172.21.15.47:0/2758512983 >> 172.21.15.47:0/2758512983 conn(0x7f81e00b7ea0 msgr2=0x7f81e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 1 -- 172.21.15.47:0/2758512983 shutdown_connections 2024-03-25T19:45:08.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.345+0000 7f81e8383640 1 -- 172.21.15.47:0/2758512983 wait complete. 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.346+0000 7f81e8383640 10 monclient: build_initial_monmap 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.346+0000 7f81e8383640 10 monclient: monmap: 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:08.349 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 librados: starting msgr at 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 librados: starting objecter 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 Processor -- start 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 -- start start 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 librados: setting wanted keys 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 librados: calling monclient init 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient: init 2024-03-25T19:45:08.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient(hunting): picked mon.noname-b con 0x7f81e013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient(hunting): picked mon.noname-c con 0x7f81e01430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b0640 con 0x7f81e00a98f0 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b0920 con 0x7f81e013f710 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00af260 con 0x7f81e01430c0 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:08.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.347+0000 7f81e8383640 10 monclient(hunting): authenticate will time out at 6223.633902s 2024-03-25T19:45:08.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.352+0000 7f81e6920640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f81e01430c0 0x7f81e00ac940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42194/0 (socket says 172.21.15.47:42194) 2024-03-25T19:45:08.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.352+0000 7f81e6920640 1 -- 172.21.15.47:0/1595245330 learned_addr learned my addr 172.21.15.47:0/1595245330 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:08.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.354+0000 7f81d6ffd640 10 client.?.objecter ms_handle_connect 0x7f81e01430c0 2024-03-25T19:45:08.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.354+0000 7f81d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.358+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495987254 0 0) 0x7f81e00af260 con 0x7f81e01430c0 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.358+0000 7f81d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:08.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.358+0000 7f81d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.358+0000 7f81d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.358+0000 7f81d6ffd640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:45:08.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.358+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c4004eb0 con 0x7f81e01430c0 2024-03-25T19:45:08.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.369+0000 7f81d6ffd640 10 client.?.objecter ms_handle_connect 0x7f81e013f710 2024-03-25T19:45:08.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.369+0000 7f81d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 10 client.?.objecter ms_handle_connect 0x7f81e00a98f0 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486511754 0 0) 0x7f81c4004eb0 con 0x7f81e01430c0 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e00af260 con 0x7f81e01430c0 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f81d8001ad0 con 0x7f81e01430c0 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.371 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.372 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:08.372 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:08.372 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:08.372 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:08.372 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:08.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.371+0000 7f81d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:08.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 171805311 0 0) 0x7f81e00af260 con 0x7f81e01430c0 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 >> v1:172.21.15.142:6789/0 conn(0x7f81e013f710 legacy=0x7f81e00abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 >> v1:172.21.15.47:6789/0 conn(0x7f81e00a98f0 legacy=0x7f81e00aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 1 monclient: found mon.c 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00ac400 con 0x7f81e01430c0 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 5 monclient: authenticate success, global_id 4682 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 10 monclient: _renew_subs 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:08.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e00af780 con 0x7f81e01430c0 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 10 client.4682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 10 monclient: _renew_subs 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e00ad230 con 0x7f81e01430c0 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.379+0000 7f81e8383640 1 librados: init done 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.380+0000 7f81e8383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.380+0000 7f81e8383640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.380+0000 7f81e8383640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.380+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e00af3f0 con 0x7f81e01430c0 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d8002020 con 0x7f81e01430c0 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f81d8004d80 con 0x7f81e01430c0 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81d6ffd640 10 monclient: dump: 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.689 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:08.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.689+0000 7f81e491c640 4 set_mon_vals no callback set 2024-03-25T19:45:08.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.690+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f81d801ad50 con 0x7f81e01430c0 2024-03-25T19:45:08.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.691+0000 7f81e611f640 10 monclient: get_auth_request con 0x7f81c406dcb0 auth_method 0 2024-03-25T19:45:08.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.691+0000 7f81d6ffd640 10 client.4682.objecter ms_handle_connect 0x7f81c406dcb0 2024-03-25T19:45:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f81d8054ab0 con 0x7f81e01430c0 2024-03-25T19:45:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 10 client.4682.objecter ms_dispatch 0x7f81e0000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:08.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 3 client.4682.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 3 client.4682.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 20 client.4682.objecter dump_active .. 0 homeless 2024-03-25T19:45:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81d80503f0 con 0x7f81e01430c0 2024-03-25T19:45:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:08.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.697+0000 7f81d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:08.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.928+0000 7f81e8383640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:08.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.928+0000 7f81e8383640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:08.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.928+0000 7f81e8383640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:08.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.928+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e00627b0 con 0x7f81e01430c0 2024-03-25T19:45:08.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.966+0000 7f81d6ffd640 1 -- 172.21.15.47:0/1595245330 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f81d80524f0 con 0x7f81e01430c0 2024-03-25T19:45:08.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.966+0000 7f81d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:08.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.966+0000 7f81d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:08.966 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:08.966 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 10 librados: watch_flush enter 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 10 librados: watch_flush exit 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 20 client.4682.objecter shutdown clearing up homeless session... 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 10 client.4682.objecter successfully canceled tick 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 >> v1:172.21.15.142:6800/1025302268 conn(0x7f81c406dcb0 legacy=0x7f81c4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 10 monclient: shutdown 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:08.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 >> v1:172.21.15.47:6790/0 conn(0x7f81e01430c0 legacy=0x7f81e00ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 shutdown_connections 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 >> 172.21.15.47:0/1595245330 conn(0x7f81e00b7ea0 msgr2=0x7f81e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 shutdown_connections 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 -- 172.21.15.47:0/1595245330 wait complete. 2024-03-25T19:45:08.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:08.969+0000 7f81e8383640 1 librados: shutdown 2024-03-25T19:45:08.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:15.788 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:15.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.903+0000 7f5458702640 10 monclient: get_monmap_and_config 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.903+0000 7f5458702640 10 monclient: build_initial_monmap 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.903+0000 7f5458702640 10 monclient: monmap: 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:15.905181+0000 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:15.905181+0000 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:15.904 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.904+0000 7f5458702640 1 Processor -- start 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.904+0000 7f5458702640 1 -- start start 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.904+0000 7f5458702640 10 monclient: init 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient(hunting): picked mon.noname-a con 0x7f54500a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient(hunting): picked mon.noname-c con 0x7f545013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient(hunting): picked mon.noname-b con 0x7f54501430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450142520 con 0x7f54500a98f0 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b28a0 con 0x7f545013f710 2024-03-25T19:45:15.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b2a30 con 0x7f54501430c0 2024-03-25T19:45:15.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:15.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.905+0000 7f5458702640 10 monclient(hunting): authenticate will time out at 6231.192215s 2024-03-25T19:45:15.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.916+0000 7f545649e640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f54500a98f0 0x7f54500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49152/0 (socket says 172.21.15.47:49152) 2024-03-25T19:45:15.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.916+0000 7f545649e640 1 -- 172.21.15.47:0/1624734771 learned_addr learned my addr 172.21.15.47:0/1624734771 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:15.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.928+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51454046 0 0) 0x7f5450142520 con 0x7f54500a98f0 2024-03-25T19:45:15.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.928+0000 7f545549c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:15.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.928+0000 7f545549c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:15.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.928+0000 7f545549c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:15.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.928+0000 7f545549c640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:45:15.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.928+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c003520 con 0x7f54500a98f0 2024-03-25T19:45:15.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.934+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496399726 0 0) 0x7f54500b28a0 con 0x7f545013f710 2024-03-25T19:45:15.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.934+0000 7f545549c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:15.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.934+0000 7f545549c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:15.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.934+0000 7f545549c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:15.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.934+0000 7f545549c640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:45:15.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.934+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450142520 con 0x7f545013f710 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1618089042 0 0) 0x7f5450142520 con 0x7f545013f710 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 >> v1:172.21.15.47:6790/0 conn(0x7f54501430c0 legacy=0x7f545014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 >> v1:172.21.15.47:6789/0 conn(0x7f54500a98f0 legacy=0x7f54500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: _finish_hunting 0 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 1 monclient: found mon.noname-c 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:45:15.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500b44c0 con 0x7f545013f710 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: _finish_auth 0 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: _check_auth_tickets 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5440001670 con 0x7f545013f710 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f545549c640 10 monclient: dump: 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:15.943 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:15.944 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:15.944 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:15.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f5458702640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:45:15.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.941+0000 7f5458702640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54400017d0 con 0x7f545013f710 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f545549c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f545549c640 1 -- 172.21.15.47:0/1624734771 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5440001be0 con 0x7f545013f710 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f545549c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f545549c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f545549c640 10 monclient: dump: 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:15.950 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f5458702640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:15.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.949+0000 7f5458702640 4 set_mon_vals no callback set 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 10 monclient: shutdown 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 1 -- 172.21.15.47:0/1624734771 >> v1:172.21.15.142:6789/0 conn(0x7f545013f710 legacy=0x7f54501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 1 -- 172.21.15.47:0/1624734771 shutdown_connections 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 1 -- 172.21.15.47:0/1624734771 >> 172.21.15.47:0/1624734771 conn(0x7f54500b7ea0 msgr2=0x7f54500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 1 -- 172.21.15.47:0/1624734771 shutdown_connections 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.950+0000 7f5458702640 1 -- 172.21.15.47:0/1624734771 wait complete. 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.951+0000 7f5458702640 10 monclient: build_initial_monmap 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.951+0000 7f5458702640 10 monclient: monmap: 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:15.953 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.951+0000 7f5458702640 1 librados: starting msgr at 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.951+0000 7f5458702640 1 librados: starting objecter 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 Processor -- start 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 -- start start 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 librados: setting wanted keys 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 librados: calling monclient init 2024-03-25T19:45:15.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient: init 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient(hunting): picked mon.noname-c con 0x7f54500a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient(hunting): picked mon.noname-b con 0x7f545013f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient(hunting): picked mon.noname-a con 0x7f54501430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54501d7590 con 0x7f54501430c0 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450138ae0 con 0x7f545013f710 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450139760 con 0x7f54500a98f0 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:15.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.952+0000 7f5458702640 10 monclient(hunting): authenticate will time out at 6231.239724s 2024-03-25T19:45:15.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.957+0000 7f5456c9f640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f54501430c0 0x7f54501e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49182/0 (socket says 172.21.15.47:49182) 2024-03-25T19:45:15.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.957+0000 7f5456c9f640 1 -- 172.21.15.47:0/2840005536 learned_addr learned my addr 172.21.15.47:0/2840005536 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:15.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.961+0000 7f5446ffd640 10 client.?.objecter ms_handle_connect 0x7f54501430c0 2024-03-25T19:45:15.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.961+0000 7f5446ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:15.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.964+0000 7f5446ffd640 10 client.?.objecter ms_handle_connect 0x7f545013f710 2024-03-25T19:45:15.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.964+0000 7f5446ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:15.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.974+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721454364 0 0) 0x7f5450138ae0 con 0x7f545013f710 2024-03-25T19:45:15.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.974+0000 7f5446ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:15.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.974+0000 7f5446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:15.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.974+0000 7f5446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:15.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.974+0000 7f5446ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:45:15.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.974+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5434004eb0 con 0x7f545013f710 2024-03-25T19:45:15.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534365050 0 0) 0x7f54501d7590 con 0x7f54501430c0 2024-03-25T19:45:15.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:15.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:15.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:15.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:45:15.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450138ae0 con 0x7f54501430c0 2024-03-25T19:45:15.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 10 client.?.objecter ms_handle_connect 0x7f54500a98f0 2024-03-25T19:45:15.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.977+0000 7f5446ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63227350 0 0) 0x7f5434004eb0 con 0x7f545013f710 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54501d7590 con 0x7f545013f710 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5440001b20 con 0x7f545013f710 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:15.991 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 719310064 0 0) 0x7f5450138ae0 con 0x7f54501430c0 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5434004eb0 con 0x7f54501430c0 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5448001a90 con 0x7f54501430c0 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.990+0000 7f5446ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:15.992 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.991+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918730836 0 0) 0x7f5450139760 con 0x7f54500a98f0 2024-03-25T19:45:15.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.991+0000 7f5446ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:15.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.991+0000 7f5446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:15.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.991+0000 7f5446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:15.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.991+0000 7f5446ffd640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:45:15.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:15.991+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450138ae0 con 0x7f54500a98f0 2024-03-25T19:45:16.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.000+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087863489 0 0) 0x7f54501d7590 con 0x7f545013f710 2024-03-25T19:45:16.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 >> v1:172.21.15.47:6790/0 conn(0x7f54500a98f0 legacy=0x7f54501383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 >> v1:172.21.15.47:6789/0 conn(0x7f54501430c0 legacy=0x7f54501e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:16.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 1 monclient: found mon.b 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54501399d0 con 0x7f545013f710 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 10 monclient: _renew_subs 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5450134de0 con 0x7f545013f710 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 10 monclient: _renew_subs 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:16.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54501d83c0 con 0x7f545013f710 2024-03-25T19:45:16.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.001+0000 7f5458702640 1 librados: init done 2024-03-25T19:45:16.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.002+0000 7f5458702640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:16.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.002+0000 7f5458702640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:16.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.002+0000 7f5458702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:16.003 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.002+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54501d7740 con 0x7f545013f710 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54400038a0 con 0x7f545013f710 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5446ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f5440003a10 con 0x7f545013f710 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5446ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5446ffd640 10 monclient: dump: 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:16.281 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.280+0000 7f5454c9b640 4 set_mon_vals no callback set 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.281+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f54400199e0 con 0x7f545013f710 2024-03-25T19:45:16.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.281+0000 7f545649e640 10 monclient: get_auth_request con 0x7f543406df50 auth_method 0 2024-03-25T19:45:16.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.281+0000 7f5446ffd640 10 client.4696.objecter ms_handle_connect 0x7f543406df50 2024-03-25T19:45:16.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f54400535f0 con 0x7f545013f710 2024-03-25T19:45:16.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 10 client.4696.objecter ms_dispatch 0x7f5450000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:16.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 3 client.4696.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:16.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 3 client.4696.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:16.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T19:45:16.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f544004f100 con 0x7f545013f710 2024-03-25T19:45:16.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:16.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.289+0000 7f5446ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:16.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.532+0000 7f5458702640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:16.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.532+0000 7f5458702640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:16.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.532+0000 7f5458702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:16.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.532+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54500627b0 con 0x7f545013f710 2024-03-25T19:45:16.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.627+0000 7f5446ffd640 1 -- 172.21.15.47:0/2840005536 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f5440051030 con 0x7f545013f710 2024-03-25T19:45:16.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.627+0000 7f5446ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:16.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.627+0000 7f5446ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:16.628 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:16.628 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:16.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 10 librados: watch_flush enter 2024-03-25T19:45:16.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 10 librados: watch_flush exit 2024-03-25T19:45:16.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T19:45:16.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 10 client.4696.objecter successfully canceled tick 2024-03-25T19:45:16.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 >> v1:172.21.15.142:6800/1025302268 conn(0x7f543406df50 legacy=0x7f5434070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 10 monclient: shutdown 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 >> v1:172.21.15.142:6789/0 conn(0x7f545013f710 legacy=0x7f54501d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 shutdown_connections 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 >> 172.21.15.47:0/2840005536 conn(0x7f54500b7ea0 msgr2=0x7f5450074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 shutdown_connections 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 -- 172.21.15.47:0/2840005536 wait complete. 2024-03-25T19:45:16.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:16.630+0000 7f5458702640 1 librados: shutdown 2024-03-25T19:45:16.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:23.448 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.542+0000 7f351dd6d640 10 monclient: get_monmap_and_config 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.542+0000 7f351dd6d640 10 monclient: build_initial_monmap 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.542+0000 7f351dd6d640 10 monclient: monmap: 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:23.543862+0000 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:23.543862+0000 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:23.542 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:23.543 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:23.543 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:23.543 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:23.543 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.542+0000 7f351dd6d640 1 Processor -- start 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.542+0000 7f351dd6d640 1 -- start start 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.542+0000 7f351dd6d640 10 monclient: init 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient(hunting): picked mon.noname-c con 0x7f35180b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient(hunting): picked mon.noname-b con 0x7f35180b1730 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient(hunting): picked mon.noname-a con 0x7f351814be90 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180af940 con 0x7f351814be90 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180b0a70 con 0x7f35180b4d60 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180b0d50 con 0x7f35180b1730 2024-03-25T19:45:23.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:23.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.543+0000 7f351dd6d640 10 monclient(hunting): authenticate will time out at 6238.830757s 2024-03-25T19:45:23.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.558+0000 7f3517fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f351814be90 0x7f351814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54502/0 (socket says 172.21.15.47:54502) 2024-03-25T19:45:23.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.558+0000 7f3517fff640 1 -- 172.21.15.47:0/3766972222 learned_addr learned my addr 172.21.15.47:0/3766972222 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:23.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.577+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792553979 0 0) 0x7f35180af940 con 0x7f351814be90 2024-03-25T19:45:23.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.577+0000 7f35167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:23.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.577+0000 7f35167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:23.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.577+0000 7f35167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:23.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.577+0000 7f35167fc640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:45:23.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.577+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3500003520 con 0x7f351814be90 2024-03-25T19:45:23.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.590+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799207891 0 0) 0x7f35180b0d50 con 0x7f35180b1730 2024-03-25T19:45:23.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.590+0000 7f35167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:23.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.590+0000 7f35167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:23.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.590+0000 7f35167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:23.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.590+0000 7f35167fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:45:23.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.590+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35180af940 con 0x7f35180b1730 2024-03-25T19:45:23.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108569457 0 0) 0x7f3500003520 con 0x7f351814be90 2024-03-25T19:45:23.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 >> v1:172.21.15.47:6790/0 conn(0x7f35180b1730 legacy=0x7f35180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:23.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 >> v1:172.21.15.142:6789/0 conn(0x7f35180b4d60 legacy=0x7f351813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:23.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:23.599 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 1 monclient: found mon.noname-a 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180b1030 con 0x7f351814be90 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f3510001a10 con 0x7f351814be90 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f351dd6d640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f351dd6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f35167fc640 10 monclient: dump: 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:23.600 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:23.601 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.598+0000 7f351dd6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3510001b70 con 0x7f351814be90 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f35167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f35167fc640 1 -- 172.21.15.47:0/3766972222 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f3510001f80 con 0x7f351814be90 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f35167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f35167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f35167fc640 10 monclient: dump: 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:23.624 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f351dd6d640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.623+0000 7f351dd6d640 4 set_mon_vals no callback set 2024-03-25T19:45:23.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.624+0000 7f351dd6d640 10 monclient: shutdown 2024-03-25T19:45:23.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.624+0000 7f351dd6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:23.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.624+0000 7f351dd6d640 1 -- 172.21.15.47:0/3766972222 >> v1:172.21.15.47:6789/0 conn(0x7f351814be90 legacy=0x7f351814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:23.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.624+0000 7f351dd6d640 1 -- 172.21.15.47:0/3766972222 shutdown_connections 2024-03-25T19:45:23.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.624+0000 7f351dd6d640 1 -- 172.21.15.47:0/3766972222 >> 172.21.15.47:0/3766972222 conn(0x7f35180617a0 msgr2=0x7f3518061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:23.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.625+0000 7f351dd6d640 1 -- 172.21.15.47:0/3766972222 shutdown_connections 2024-03-25T19:45:23.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.625+0000 7f351dd6d640 1 -- 172.21.15.47:0/3766972222 wait complete. 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.626+0000 7f351dd6d640 10 monclient: build_initial_monmap 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.626+0000 7f351dd6d640 10 monclient: monmap: 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:23.627 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.626+0000 7f351dd6d640 1 librados: starting msgr at 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.626+0000 7f351dd6d640 1 librados: starting objecter 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.626+0000 7f351dd6d640 1 Processor -- start 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 1 -- start start 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 1 librados: setting wanted keys 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 1 librados: calling monclient init 2024-03-25T19:45:23.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 10 monclient: init 2024-03-25T19:45:23.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:23.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:23.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 10 monclient(hunting): picked mon.noname-a con 0x7f35180b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:23.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.627+0000 7f351dd6d640 10 monclient(hunting): picked mon.noname-b con 0x7f35180b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:23.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.628+0000 7f351dd6d640 10 monclient(hunting): picked mon.noname-c con 0x7f351814be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:23.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.628+0000 7f351dd6d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35181383e0 con 0x7f35180b1730 2024-03-25T19:45:23.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.628+0000 7f351dd6d640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35181d8590 con 0x7f35180b4d60 2024-03-25T19:45:23.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.628+0000 7f351dd6d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35181d8720 con 0x7f351814be90 2024-03-25T19:45:23.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.628+0000 7f351dd6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:23.630 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.628+0000 7f351dd6d640 10 monclient(hunting): authenticate will time out at 6238.915166s 2024-03-25T19:45:23.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.641+0000 7f3516ffd640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f35180b4d60 0x7f35181d7e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:40808/0 (socket says 172.21.15.47:40808) 2024-03-25T19:45:23.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.641+0000 7f3516ffd640 1 -- 172.21.15.47:0/2656940369 learned_addr learned my addr 172.21.15.47:0/2656940369 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:23.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.642+0000 7f34fffff640 10 client.?.objecter ms_handle_connect 0x7f35180b4d60 2024-03-25T19:45:23.642 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.642+0000 7f34fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:23.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.649+0000 7f34fffff640 10 client.?.objecter ms_handle_connect 0x7f351814be90 2024-03-25T19:45:23.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.649+0000 7f34fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:23.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.654+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560504759 0 0) 0x7f35181d8590 con 0x7f35180b4d60 2024-03-25T19:45:23.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.654+0000 7f34fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:23.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.654+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:23.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.654+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:23.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.654+0000 7f34fffff640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:45:23.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.654+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3504004eb0 con 0x7f35180b4d60 2024-03-25T19:45:23.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.661+0000 7f34fffff640 10 client.?.objecter ms_handle_connect 0x7f35180b1730 2024-03-25T19:45:23.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.661+0000 7f34fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:23.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.664+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384029084 0 0) 0x7f35181d8720 con 0x7f351814be90 2024-03-25T19:45:23.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.664+0000 7f34fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:23.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.664+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:23.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.664+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:23.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.664+0000 7f34fffff640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:45:23.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.664+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35181d8590 con 0x7f351814be90 2024-03-25T19:45:23.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.673+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468950093 0 0) 0x7f35181383e0 con 0x7f35180b1730 2024-03-25T19:45:23.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.673+0000 7f34fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:23.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.673+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:23.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.673+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:23.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.673+0000 7f34fffff640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:45:23.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.673+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35181d8720 con 0x7f35180b1730 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.680+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4974533 0 0) 0x7f3504004eb0 con 0x7f35180b4d60 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.681+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35181383e0 con 0x7f35180b4d60 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.681+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f350c001a10 con 0x7f35180b4d60 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.681+0000 7f34fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.681+0000 7f34fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.681+0000 7f34fffff640 10 monclient(hunting): dump: 2024-03-25T19:45:23.681 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:23.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.681+0000 7f34fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.688+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2225699931 0 0) 0x7f35181d8720 con 0x7f35180b1730 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.688+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3504004eb0 con 0x7f35180b1730 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.688+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f35080017d0 con 0x7f35180b1730 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.688+0000 7f34fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.688+0000 7f34fffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.688+0000 7f34fffff640 10 monclient(hunting): dump: 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:23.689 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:23.690 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 170283291 0 0) 0x7f3504004eb0 con 0x7f35180b1730 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 >> v1:172.21.15.47:6790/0 conn(0x7f351814be90 legacy=0x7f351812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 >> v1:172.21.15.142:6789/0 conn(0x7f35180b4d60 legacy=0x7f35181d7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 1 monclient: found mon.a 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3518143900 con 0x7f35180b1730 2024-03-25T19:45:23.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 10 monclient: _finish_auth 0 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f34fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 monclient: _renew_subs 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3518130490 con 0x7f35180b1730 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 monclient: _renew_subs 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3518136420 con 0x7f35180b1730 2024-03-25T19:45:23.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 1 librados: init done 2024-03-25T19:45:23.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:23.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:23.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:23.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:23.695+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35181401f0 con 0x7f35180b1730 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3508001cc0 con 0x7f35180b1730 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f34fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f3508003790 con 0x7f35180b1730 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f34fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f34fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f34fffff640 10 monclient: dump: 2024-03-25T19:45:24.017 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:24.018 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.016+0000 7f3515ffb640 4 set_mon_vals no callback set 2024-03-25T19:45:24.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.017+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f3508019760 con 0x7f35180b1730 2024-03-25T19:45:24.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.018+0000 7f3516ffd640 10 monclient: get_auth_request con 0x7f350406df70 auth_method 0 2024-03-25T19:45:24.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.018+0000 7f34fffff640 10 client.4707.objecter ms_handle_connect 0x7f350406df70 2024-03-25T19:45:24.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f35080534e0 con 0x7f35180b1730 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 10 client.4707.objecter ms_dispatch 0x7f3518000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 3 client.4707.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 3 client.4707.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f350804eff0 con 0x7f35180b1730 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:24.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.034+0000 7f34fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:24.188 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:01.423 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.194 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:01.429 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.253 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:01.489 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.265+0000 7f351dd6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:24.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.265+0000 7f351dd6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:24.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.265+0000 7f351dd6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:24.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.265+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3518136720 con 0x7f35180b1730 2024-03-25T19:45:24.272 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:01.508 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.280 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:01.516 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.283 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:01.473 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.288 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:01.478 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.308 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:01.498 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.313 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:01.549 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.363+0000 7f34fffff640 1 -- 172.21.15.47:0/2656940369 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f3518136720 con 0x7f35180b1730 2024-03-25T19:45:24.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.363+0000 7f34fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:24.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.363+0000 7f34fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:24.363 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:24.363 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:24.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.365+0000 7f351dd6d640 10 librados: watch_flush enter 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.365+0000 7f351dd6d640 10 librados: watch_flush exit 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.365+0000 7f351dd6d640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.365+0000 7f351dd6d640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 >> v1:172.21.15.142:6800/1025302268 conn(0x7f350406df70 legacy=0x7f3504070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 10 monclient: shutdown 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 >> v1:172.21.15.47:6789/0 conn(0x7f35180b1730 legacy=0x7f3518136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 shutdown_connections 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 >> 172.21.15.47:0/2656940369 conn(0x7f35180617a0 msgr2=0x7f351813c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 shutdown_connections 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 -- 172.21.15.47:0/2656940369 wait complete. 2024-03-25T19:45:24.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:24.366+0000 7f351dd6d640 1 librados: shutdown 2024-03-25T19:45:24.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:31.185 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:31.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.301+0000 7f4ae0c15640 10 monclient: get_monmap_and_config 2024-03-25T19:45:31.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.301+0000 7f4ae0c15640 10 monclient: build_initial_monmap 2024-03-25T19:45:31.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.301+0000 7f4ae0c15640 10 monclient: monmap: 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:31.302729+0000 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:31.302729+0000 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:31.302 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:31.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.302+0000 7f4ae0c15640 1 Processor -- start 2024-03-25T19:45:31.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.302+0000 7f4ae0c15640 1 -- start start 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.302+0000 7f4ae0c15640 10 monclient: init 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient(hunting): picked mon.noname-a con 0x7f4adc0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient(hunting): picked mon.noname-b con 0x7f4adc13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient(hunting): picked mon.noname-c con 0x7f4adc1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc142520 con 0x7f4adc0a98f0 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc0b28a0 con 0x7f4adc1430c0 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc0b2a30 con 0x7f4adc13f710 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:31.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.303+0000 7f4ae0c15640 10 monclient(hunting): authenticate will time out at 6246.589572s 2024-03-25T19:45:31.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.314+0000 7f4adad9d640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4adc1430c0 0x7f4adc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:48064/0 (socket says 172.21.15.47:48064) 2024-03-25T19:45:31.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.314+0000 7f4adad9d640 1 -- 172.21.15.47:0/8919161 learned_addr learned my addr 172.21.15.47:0/8919161 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:31.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.332+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510243728 0 0) 0x7f4adc0b28a0 con 0x7f4adc1430c0 2024-03-25T19:45:31.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.332+0000 7f4ad9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:31.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.332+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:31.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.332+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:31.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.332+0000 7f4ad9d9b640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:45:31.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.332+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac8003520 con 0x7f4adc1430c0 2024-03-25T19:45:31.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.342+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545259017 0 0) 0x7f4adc142520 con 0x7f4adc0a98f0 2024-03-25T19:45:31.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.342+0000 7f4ad9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:31.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.342+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:31.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.342+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:31.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.342+0000 7f4ad9d9b640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:45:31.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.342+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc0b28a0 con 0x7f4adc0a98f0 2024-03-25T19:45:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.353+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973673259 0 0) 0x7f4adc0b2a30 con 0x7f4adc13f710 2024-03-25T19:45:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.353+0000 7f4ad9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.353+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.353+0000 7f4ad9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.353+0000 7f4ad9d9b640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:45:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.353+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc142520 con 0x7f4adc13f710 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.354+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496190576 0 0) 0x7f4adc0b28a0 con 0x7f4adc0a98f0 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.354+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 >> v1:172.21.15.47:6790/0 conn(0x7f4adc13f710 legacy=0x7f4adc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.354+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 >> v1:172.21.15.142:6789/0 conn(0x7f4adc1430c0 legacy=0x7f4adc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.354+0000 7f4ad9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.354+0000 7f4ad9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc0b44c0 con 0x7f4adc0a98f0 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4acc001670 con 0x7f4adc0a98f0 2024-03-25T19:45:31.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ad9d9b640 10 monclient: dump: 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ae0c15640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:45:31.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.355+0000 7f4ae0c15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:31.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4acc0017d0 con 0x7f4adc0a98f0 2024-03-25T19:45:31.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ad9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:31.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ad9d9b640 1 -- 172.21.15.47:0/8919161 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4acc001be0 con 0x7f4adc0a98f0 2024-03-25T19:45:31.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:31.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 4 set_mon_vals no callback set 2024-03-25T19:45:31.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ad9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ad9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ad9d9b640 10 monclient: dump: 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:31.364 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 10 monclient: shutdown 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 1 -- 172.21.15.47:0/8919161 >> v1:172.21.15.47:6789/0 conn(0x7f4adc0a98f0 legacy=0x7f4adc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 1 -- 172.21.15.47:0/8919161 shutdown_connections 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.364+0000 7f4ae0c15640 1 -- 172.21.15.47:0/8919161 >> 172.21.15.47:0/8919161 conn(0x7f4adc0b7ea0 msgr2=0x7f4adc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.365+0000 7f4ae0c15640 1 -- 172.21.15.47:0/8919161 shutdown_connections 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.365+0000 7f4ae0c15640 1 -- 172.21.15.47:0/8919161 wait complete. 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.365+0000 7f4ae0c15640 10 monclient: build_initial_monmap 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.365+0000 7f4ae0c15640 10 monclient: monmap: 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:31.367 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 1 librados: starting msgr at 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 1 librados: starting objecter 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 1 Processor -- start 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 1 -- start start 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 1 librados: setting wanted keys 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 1 librados: calling monclient init 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.366+0000 7f4ae0c15640 10 monclient: init 2024-03-25T19:45:31.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient(hunting): picked mon.noname-c con 0x7f4adc0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient(hunting): picked mon.noname-b con 0x7f4adc13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient(hunting): picked mon.noname-a con 0x7f4adc1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc0b16e0 con 0x7f4adc1430c0 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc1dabb0 con 0x7f4adc13f710 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc1da880 con 0x7f4adc0a98f0 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:31.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.367+0000 7f4ae0c15640 10 monclient(hunting): authenticate will time out at 6246.653369s 2024-03-25T19:45:31.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.378+0000 7f4adad9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4adc1430c0 0x7f4adc1e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54854/0 (socket says 172.21.15.47:54854) 2024-03-25T19:45:31.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.378+0000 7f4adad9d640 1 -- 172.21.15.47:0/1547619480 learned_addr learned my addr 172.21.15.47:0/1547619480 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:31.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.379+0000 7f4ad2ffd640 10 client.?.objecter ms_handle_connect 0x7f4adc1430c0 2024-03-25T19:45:31.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.379+0000 7f4ad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:31.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.384+0000 7f4ad2ffd640 10 client.?.objecter ms_handle_connect 0x7f4adc13f710 2024-03-25T19:45:31.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.384+0000 7f4ad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:31.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.387+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688455247 0 0) 0x7f4adc0b16e0 con 0x7f4adc1430c0 2024-03-25T19:45:31.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.387+0000 7f4ad2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:31.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.387+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:31.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.387+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:31.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.387+0000 7f4ad2ffd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:45:31.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.387+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4abc004eb0 con 0x7f4adc1430c0 2024-03-25T19:45:31.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.393+0000 7f4ad2ffd640 10 client.?.objecter ms_handle_connect 0x7f4adc0a98f0 2024-03-25T19:45:31.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.393+0000 7f4ad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:31.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.408+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736188365 0 0) 0x7f4adc1dabb0 con 0x7f4adc13f710 2024-03-25T19:45:31.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.408+0000 7f4ad2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:31.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.408+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:31.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.408+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:31.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.408+0000 7f4ad2ffd640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:45:31.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.408+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc0b16e0 con 0x7f4adc13f710 2024-03-25T19:45:31.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.416+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359862535 0 0) 0x7f4abc004eb0 con 0x7f4adc1430c0 2024-03-25T19:45:31.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.417+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4adc1dabb0 con 0x7f4adc1430c0 2024-03-25T19:45:31.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.417+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ad4001b30 con 0x7f4adc1430c0 2024-03-25T19:45:31.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.417+0000 7f4ad2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:31.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.417+0000 7f4ad2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.417+0000 7f4ad2ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:31.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.417+0000 7f4ad2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:31.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.418+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626053946 0 0) 0x7f4adc1da880 con 0x7f4adc0a98f0 2024-03-25T19:45:31.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.418+0000 7f4ad2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:31.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.418+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:31.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.418+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:31.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.418+0000 7f4ad2ffd640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:45:31.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.418+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4abc004eb0 con 0x7f4adc0a98f0 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 84081204 0 0) 0x7f4adc1dabb0 con 0x7f4adc1430c0 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 >> v1:172.21.15.47:6790/0 conn(0x7f4adc0a98f0 legacy=0x7f4adc1db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 >> v1:172.21.15.142:6789/0 conn(0x7f4adc13f710 legacy=0x7f4adc0b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 1 monclient: found mon.a 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:31.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc14eaf0 con 0x7f4adc1430c0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 10 monclient: _renew_subs 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4adc14e050 con 0x7f4adc1430c0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 10 monclient: _renew_subs 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc14e3f0 con 0x7f4adc1430c0 2024-03-25T19:45:31.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.426+0000 7f4ae0c15640 1 librados: init done 2024-03-25T19:45:31.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.427+0000 7f4ae0c15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:31.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.427+0000 7f4ae0c15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:31.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.427+0000 7f4ae0c15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:31.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.427+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc0b0c90 con 0x7f4adc1430c0 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ad4001fe0 con 0x7f4adc1430c0 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f4ad4004d00 con 0x7f4adc1430c0 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad2ffd640 10 monclient: dump: 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:31.448 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.448+0000 7f4ad959a640 4 set_mon_vals no callback set 2024-03-25T19:45:31.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.449+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f4ad4004f20 con 0x7f4adc1430c0 2024-03-25T19:45:31.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.449+0000 7f4ada59c640 10 monclient: get_auth_request con 0x7f4abc06dc10 auth_method 0 2024-03-25T19:45:31.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.450+0000 7f4ad2ffd640 10 client.4713.objecter ms_handle_connect 0x7f4abc06dc10 2024-03-25T19:45:31.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.452+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f4ad4054a70 con 0x7f4adc1430c0 2024-03-25T19:45:31.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.452+0000 7f4ad2ffd640 10 client.4713.objecter ms_dispatch 0x7f4adc000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:31.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.452+0000 7f4ad2ffd640 3 client.4713.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:31.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.452+0000 7f4ad2ffd640 3 client.4713.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:31.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.452+0000 7f4ad2ffd640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:45:31.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.772+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ad4050580 con 0x7f4adc1430c0 2024-03-25T19:45:31.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.772+0000 7f4ad2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:31.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:31.772+0000 7f4ad2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:32.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.023+0000 7f4ae0c15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:32.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.024+0000 7f4ae0c15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:32.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.024+0000 7f4ae0c15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:32.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.024+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4adc0627b0 con 0x7f4adc1430c0 2024-03-25T19:45:32.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.047+0000 7f4ad2ffd640 1 -- 172.21.15.47:0/1547619480 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f4ad40524b0 con 0x7f4adc1430c0 2024-03-25T19:45:32.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.047+0000 7f4ad2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:32.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.047+0000 7f4ad2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:32.047 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:32.047 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 10 librados: watch_flush enter 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 10 librados: watch_flush exit 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 >> v1:172.21.15.142:6800/1025302268 conn(0x7f4abc06dc10 legacy=0x7f4abc0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 10 monclient: shutdown 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.049+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 >> v1:172.21.15.47:6789/0 conn(0x7f4adc1430c0 legacy=0x7f4adc1e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.050+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 shutdown_connections 2024-03-25T19:45:32.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.050+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 >> 172.21.15.47:0/1547619480 conn(0x7f4adc0b7ea0 msgr2=0x7f4adc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:32.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.050+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 shutdown_connections 2024-03-25T19:45:32.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.050+0000 7f4ae0c15640 1 -- 172.21.15.47:0/1547619480 wait complete. 2024-03-25T19:45:32.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:32.050+0000 7f4ae0c15640 1 librados: shutdown 2024-03-25T19:45:32.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:38.867 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:38.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.986+0000 7fce66eba640 10 monclient: get_monmap_and_config 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.986+0000 7fce66eba640 10 monclient: build_initial_monmap 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.986+0000 7fce66eba640 10 monclient: monmap: 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:38.987331+0000 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:38.987331+0000 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:38.986 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:38.987 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.987+0000 7fce66eba640 1 Processor -- start 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.987+0000 7fce66eba640 1 -- start start 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.987+0000 7fce66eba640 10 monclient: init 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.987+0000 7fce66eba640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.987+0000 7fce66eba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.987+0000 7fce66eba640 10 monclient(hunting): picked mon.noname-c con 0x7fce600b4d60 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 10 monclient(hunting): picked mon.noname-a con 0x7fce600b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:38.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 10 monclient(hunting): picked mon.noname-b con 0x7fce6014be90 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:38.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600af940 con 0x7fce600b1730 2024-03-25T19:45:38.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600b0a70 con 0x7fce600b4d60 2024-03-25T19:45:38.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600b0d50 con 0x7fce6014be90 2024-03-25T19:45:38.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:38.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.988+0000 7fce66eba640 10 monclient(hunting): authenticate will time out at 6254.274014s 2024-03-25T19:45:38.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.995+0000 7fce64c56640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fce600b4d60 0x7fce6013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:54724/0 (socket says 172.21.15.47:54724) 2024-03-25T19:45:38.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:38.995+0000 7fce64c56640 1 -- 172.21.15.47:0/1122370383 learned_addr learned my addr 172.21.15.47:0/1122370383 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:39.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.007+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948842563 0 0) 0x7fce600b0d50 con 0x7fce6014be90 2024-03-25T19:45:39.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.008+0000 7fce5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:39.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.008+0000 7fce5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:39.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.008+0000 7fce5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:39.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.008+0000 7fce5f7fe640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:45:39.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.008+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50003520 con 0x7fce6014be90 2024-03-25T19:45:39.011 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.012+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870567350 0 0) 0x7fce600af940 con 0x7fce600b1730 2024-03-25T19:45:39.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.012+0000 7fce5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:39.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.012+0000 7fce5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:39.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.012+0000 7fce5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:39.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.012+0000 7fce5f7fe640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:45:39.012 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.012+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce600b0d50 con 0x7fce600b1730 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4234336863 0 0) 0x7fce50003520 con 0x7fce6014be90 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 >> v1:172.21.15.142:6789/0 conn(0x7fce600b4d60 legacy=0x7fce6013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 >> v1:172.21.15.47:6789/0 conn(0x7fce600b1730 legacy=0x7fce600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600b1030 con 0x7fce6014be90 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fce58001a10 con 0x7fce6014be90 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:39.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce5f7fe640 10 monclient: dump: 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce66eba640 5 monclient: authenticate success, global_id 4700 2024-03-25T19:45:39.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.013+0000 7fce66eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce58001b70 con 0x7fce6014be90 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce5f7fe640 1 -- 172.21.15.47:0/1122370383 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fce58001f80 con 0x7fce6014be90 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce5f7fe640 10 monclient: dump: 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.026 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce66eba640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:39.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.026+0000 7fce66eba640 4 set_mon_vals no callback set 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 10 monclient: shutdown 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 1 -- 172.21.15.47:0/1122370383 >> v1:172.21.15.47:6790/0 conn(0x7fce6014be90 legacy=0x7fce6014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 1 -- 172.21.15.47:0/1122370383 shutdown_connections 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 1 -- 172.21.15.47:0/1122370383 >> 172.21.15.47:0/1122370383 conn(0x7fce600617a0 msgr2=0x7fce60061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 1 -- 172.21.15.47:0/1122370383 shutdown_connections 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.027+0000 7fce66eba640 1 -- 172.21.15.47:0/1122370383 wait complete. 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.028+0000 7fce66eba640 10 monclient: build_initial_monmap 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.028+0000 7fce66eba640 10 monclient: monmap: 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:39.029 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 1 librados: starting msgr at 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 1 librados: starting objecter 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 1 Processor -- start 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 1 -- start start 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 1 librados: setting wanted keys 2024-03-25T19:45:39.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 1 librados: calling monclient init 2024-03-25T19:45:39.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.029+0000 7fce66eba640 10 monclient: init 2024-03-25T19:45:39.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:39.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:39.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient(hunting): picked mon.noname-a con 0x7fce600b1730 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient(hunting): picked mon.noname-c con 0x7fce600b4d60 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient(hunting): picked mon.noname-b con 0x7fce6014be90 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce601421e0 con 0x7fce600b1730 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce60142410 con 0x7fce6014be90 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce60143040 con 0x7fce600b4d60 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:39.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.030+0000 7fce66eba640 10 monclient(hunting): authenticate will time out at 6254.316235s 2024-03-25T19:45:39.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.037+0000 7fce65457640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fce6014be90 0x7fce60136ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:54782/0 (socket says 172.21.15.47:54782) 2024-03-25T19:45:39.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.037+0000 7fce65457640 1 -- 172.21.15.47:0/2294435848 learned_addr learned my addr 172.21.15.47:0/2294435848 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:39.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.039+0000 7fce5d7fa640 10 client.?.objecter ms_handle_connect 0x7fce6014be90 2024-03-25T19:45:39.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.039+0000 7fce5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:39.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.048+0000 7fce5d7fa640 10 client.?.objecter ms_handle_connect 0x7fce600b1730 2024-03-25T19:45:39.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.048+0000 7fce5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:39.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.050+0000 7fce5d7fa640 10 client.?.objecter ms_handle_connect 0x7fce600b4d60 2024-03-25T19:45:39.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.050+0000 7fce5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:39.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.059+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294351425 0 0) 0x7fce60142410 con 0x7fce6014be90 2024-03-25T19:45:39.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.059+0000 7fce5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:39.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.059+0000 7fce5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:39.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.059+0000 7fce5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:39.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.059+0000 7fce5d7fa640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:45:39.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.059+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce44004eb0 con 0x7fce6014be90 2024-03-25T19:45:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.074+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972508436 0 0) 0x7fce60143040 con 0x7fce600b4d60 2024-03-25T19:45:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.074+0000 7fce5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.074+0000 7fce5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.074+0000 7fce5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.074+0000 7fce5d7fa640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:45:39.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.074+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce60142410 con 0x7fce600b4d60 2024-03-25T19:45:39.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.084+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361971745 0 0) 0x7fce601421e0 con 0x7fce600b1730 2024-03-25T19:45:39.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.084+0000 7fce5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:39.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.084+0000 7fce5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:39.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.084+0000 7fce5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:39.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.084+0000 7fce5d7fa640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:45:39.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.084+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce60143040 con 0x7fce600b1730 2024-03-25T19:45:39.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1472289198 0 0) 0x7fce44004eb0 con 0x7fce6014be90 2024-03-25T19:45:39.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce601421e0 con 0x7fce6014be90 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fce58001f80 con 0x7fce6014be90 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 10 monclient(hunting): dump: 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:39.087 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:39.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.087+0000 7fce5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:39.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.102+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073745946 0 0) 0x7fce60143040 con 0x7fce600b1730 2024-03-25T19:45:39.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.103+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce44004eb0 con 0x7fce600b1730 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.103+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fce54001860 con 0x7fce600b1730 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.103+0000 7fce5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.103+0000 7fce5d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.103+0000 7fce5d7fa640 10 monclient(hunting): dump: 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:39.103 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:39.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 580889187 0 0) 0x7fce601421e0 con 0x7fce6014be90 2024-03-25T19:45:39.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 >> v1:172.21.15.47:6790/0 conn(0x7fce600b4d60 legacy=0x7fce60142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 >> v1:172.21.15.47:6789/0 conn(0x7fce600b1730 legacy=0x7fce601405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 1 monclient: found mon.b 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce601388e0 con 0x7fce6014be90 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 10 monclient: _renew_subs 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:39.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce601375f0 con 0x7fce6014be90 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 10 monclient: _renew_subs 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce601380d0 con 0x7fce6014be90 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.105+0000 7fce66eba640 1 librados: init done 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.106+0000 7fce66eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.106+0000 7fce66eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.106+0000 7fce66eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:39.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.106+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce60142630 con 0x7fce6014be90 2024-03-25T19:45:39.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce58003c20 con 0x7fce6014be90 2024-03-25T19:45:39.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:39.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fce58003ff0 con 0x7fce6014be90 2024-03-25T19:45:39.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:39.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5effd640 4 set_mon_vals no callback set 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.396+0000 7fce5d7fa640 10 monclient: dump: 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:39.397 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:39.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.397+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fce58004210 con 0x7fce6014be90 2024-03-25T19:45:39.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.398+0000 7fce64c56640 10 monclient: get_auth_request con 0x7fce4406df70 auth_method 0 2024-03-25T19:45:39.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.398+0000 7fce5d7fa640 10 client.4711.objecter ms_handle_connect 0x7fce4406df70 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.402+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7fce58054b10 con 0x7fce6014be90 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.402+0000 7fce5d7fa640 10 client.4711.objecter ms_dispatch 0x7fce60000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.402+0000 7fce5d7fa640 3 client.4711.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.402+0000 7fce5d7fa640 3 client.4711.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.403+0000 7fce5d7fa640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.403+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce58050620 con 0x7fce6014be90 2024-03-25T19:45:39.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.403+0000 7fce5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:39.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.403+0000 7fce5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:39.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.660+0000 7fce66eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:39.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.660+0000 7fce66eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:39.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.660+0000 7fce66eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:45:39.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.660+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce60138a70 con 0x7fce6014be90 2024-03-25T19:45:39.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.697+0000 7fce5d7fa640 1 -- 172.21.15.47:0/2294435848 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7fce58052550 con 0x7fce6014be90 2024-03-25T19:45:39.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.697+0000 7fce5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:39.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.697+0000 7fce5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:39.697 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:39.697 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 10 librados: watch_flush enter 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 10 librados: watch_flush exit 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 >> v1:172.21.15.142:6800/1025302268 conn(0x7fce4406df70 legacy=0x7fce44070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 10 monclient: shutdown 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 >> v1:172.21.15.142:6789/0 conn(0x7fce6014be90 legacy=0x7fce60136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 shutdown_connections 2024-03-25T19:45:39.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 >> 172.21.15.47:0/2294435848 conn(0x7fce600617a0 msgr2=0x7fce600b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:39.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 shutdown_connections 2024-03-25T19:45:39.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 -- 172.21.15.47:0/2294435848 wait complete. 2024-03-25T19:45:39.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:39.700+0000 7fce66eba640 1 librados: shutdown 2024-03-25T19:45:39.710 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:43.500 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:20.735 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.507 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:20.743 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.515 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:20.705 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.516 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:20.752 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.518 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:20.708 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.519 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:20.754 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.539 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:20.729 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.540 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:20.776 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.579 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:20.814 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.156 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:21.391 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.226 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:21.461 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.232 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:21.422 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.238 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:21.473 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.240 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:21.430 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.254 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:21.490 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.298 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:21.534 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.304 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:21.494 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:44.326 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:21.562 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:46.517 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:46.633 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.634+0000 7ff803dfe640 10 monclient: get_monmap_and_config 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.634+0000 7ff803dfe640 10 monclient: build_initial_monmap 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.634+0000 7ff803dfe640 10 monclient: monmap: 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:46.634949+0000 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:46.634949+0000 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:46.634 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.634+0000 7ff803dfe640 1 Processor -- start 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.635+0000 7ff803dfe640 1 -- start start 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.635+0000 7ff803dfe640 10 monclient: init 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc0a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc13f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc1430c0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:46.636 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc142520 con 0x7ff7fc1430c0 2024-03-25T19:45:46.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0b28a0 con 0x7ff7fc13f710 2024-03-25T19:45:46.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0b2a30 con 0x7ff7fc0a98f0 2024-03-25T19:45:46.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:46.637 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.636+0000 7ff803dfe640 10 monclient(hunting): authenticate will time out at 6261.922367s 2024-03-25T19:45:46.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.639+0000 7ff801399640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff7fc13f710 0x7ff7fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:59186/0 (socket says 172.21.15.47:59186) 2024-03-25T19:45:46.638 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.639+0000 7ff801399640 1 -- 172.21.15.47:0/1042471876 learned_addr learned my addr 172.21.15.47:0/1042471876 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:46.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.651+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725260799 0 0) 0x7ff7fc0b28a0 con 0x7ff7fc13f710 2024-03-25T19:45:46.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.651+0000 7ff800b98640 10 monclient(hunting): _negotiate 2024-03-25T19:45:46.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.651+0000 7ff800b98640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:46.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.651+0000 7ff800b98640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:46.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.651+0000 7ff800b98640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:45:46.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.651+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e8003520 con 0x7ff7fc13f710 2024-03-25T19:45:46.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.654+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029749270 0 0) 0x7ff7fc0b2a30 con 0x7ff7fc0a98f0 2024-03-25T19:45:46.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.654+0000 7ff800b98640 10 monclient(hunting): _negotiate 2024-03-25T19:45:46.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.654+0000 7ff800b98640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:46.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.654+0000 7ff800b98640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:46.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.654+0000 7ff800b98640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:45:46.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.654+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7fc0b28a0 con 0x7ff7fc0a98f0 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815746674 0 0) 0x7ff7e8003520 con 0x7ff7fc13f710 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 >> v1:172.21.15.47:6790/0 conn(0x7ff7fc0a98f0 legacy=0x7ff7fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 >> v1:172.21.15.47:6789/0 conn(0x7ff7fc1430c0 legacy=0x7ff7fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: _finish_hunting 0 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 1 monclient: found mon.noname-c 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:45:46.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0b44c0 con 0x7ff7fc13f710 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: _finish_auth 0 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: _check_auth_tickets 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff7ec001690 con 0x7ff7fc13f710 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff803dfe640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff803dfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff800b98640 10 monclient: dump: 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:46.662 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:46.663 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:46.663 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:46.663 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:46.663 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:46.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.661+0000 7ff803dfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:46.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7ec0017f0 con 0x7ff7fc13f710 2024-03-25T19:45:46.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff800b98640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:46.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff800b98640 1 -- 172.21.15.47:0/1042471876 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff7ec001c00 con 0x7ff7fc13f710 2024-03-25T19:45:46.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff800b98640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:46.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff800b98640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:46.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff800b98640 10 monclient: dump: 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff803dfe640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:46.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.672+0000 7ff803dfe640 4 set_mon_vals no callback set 2024-03-25T19:45:46.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 10 monclient: shutdown 2024-03-25T19:45:46.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:46.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 1 -- 172.21.15.47:0/1042471876 >> v1:172.21.15.142:6789/0 conn(0x7ff7fc13f710 legacy=0x7ff7fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 1 -- 172.21.15.47:0/1042471876 shutdown_connections 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 1 -- 172.21.15.47:0/1042471876 >> 172.21.15.47:0/1042471876 conn(0x7ff7fc0b7ea0 msgr2=0x7ff7fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 1 -- 172.21.15.47:0/1042471876 shutdown_connections 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.673+0000 7ff803dfe640 1 -- 172.21.15.47:0/1042471876 wait complete. 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 10 monclient: build_initial_monmap 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 10 monclient: monmap: 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:46.675 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 1 librados: starting msgr at 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 1 librados: starting objecter 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 1 Processor -- start 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 1 -- start start 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 1 librados: setting wanted keys 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 1 librados: calling monclient init 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 10 monclient: init 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.674+0000 7ff803dfe640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc0a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc13f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc1430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc1d77a0 con 0x7ff7fc0a98f0 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc138e40 con 0x7ff7fc1430c0 2024-03-25T19:45:46.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0aba30 con 0x7ff7fc13f710 2024-03-25T19:45:46.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:46.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.675+0000 7ff803dfe640 10 monclient(hunting): authenticate will time out at 6261.961070s 2024-03-25T19:45:46.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.681+0000 7ff801b9a640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff7fc0a98f0 0x7ff7fc138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47218/0 (socket says 172.21.15.47:47218) 2024-03-25T19:45:46.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.681+0000 7ff801b9a640 1 -- 172.21.15.47:0/355980238 learned_addr learned my addr 172.21.15.47:0/355980238 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:46.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.686+0000 7ff7f27fc640 10 client.?.objecter ms_handle_connect 0x7ff7fc0a98f0 2024-03-25T19:45:46.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.686+0000 7ff7f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:46.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.691+0000 7ff7f27fc640 10 client.?.objecter ms_handle_connect 0x7ff7fc1430c0 2024-03-25T19:45:46.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.691+0000 7ff7f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:46.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.691+0000 7ff7f27fc640 10 client.?.objecter ms_handle_connect 0x7ff7fc13f710 2024-03-25T19:45:46.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.691+0000 7ff7f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:46.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.695+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639100724 0 0) 0x7ff7fc0aba30 con 0x7ff7fc13f710 2024-03-25T19:45:46.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.695+0000 7ff7f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:46.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.695+0000 7ff7f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:46.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.695+0000 7ff7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:46.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.695+0000 7ff7f27fc640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:45:46.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.695+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0004eb0 con 0x7ff7fc13f710 2024-03-25T19:45:46.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.703+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103230475 0 0) 0x7ff7fc1d77a0 con 0x7ff7fc0a98f0 2024-03-25T19:45:46.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.704+0000 7ff7f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:46.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.704+0000 7ff7f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:46.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.704+0000 7ff7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:46.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.704+0000 7ff7f27fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:45:46.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.704+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7fc0aba30 con 0x7ff7fc0a98f0 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913961376 0 0) 0x7ff7e0004eb0 con 0x7ff7fc13f710 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7fc1d77a0 con 0x7ff7fc13f710 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff7ec001b00 con 0x7ff7fc13f710 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 10 monclient(hunting): dump: 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:46.706 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:46.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.706+0000 7ff7f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:46.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.711+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152715053 0 0) 0x7ff7fc138e40 con 0x7ff7fc1430c0 2024-03-25T19:45:46.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.711+0000 7ff7f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:46.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.711+0000 7ff7f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:46.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.711+0000 7ff7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:46.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.711+0000 7ff7f27fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:45:46.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.711+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0004eb0 con 0x7ff7fc1430c0 2024-03-25T19:45:46.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3730181367 0 0) 0x7ff7fc1d77a0 con 0x7ff7fc13f710 2024-03-25T19:45:46.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 >> v1:172.21.15.142:6789/0 conn(0x7ff7fc1430c0 legacy=0x7ff7fc134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 >> v1:172.21.15.47:6789/0 conn(0x7ff7fc0a98f0 legacy=0x7ff7fc138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:46.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 1 monclient: found mon.c 2024-03-25T19:45:46.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc1d8340 con 0x7ff7fc13f710 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff803dfe640 5 monclient: authenticate success, global_id 4709 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff803dfe640 10 monclient: _renew_subs 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff803dfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.713+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc134820 con 0x7ff7fc13f710 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.714+0000 7ff803dfe640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.714+0000 7ff803dfe640 10 monclient: _renew_subs 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.714+0000 7ff803dfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.714+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc1380c0 con 0x7ff7fc13f710 2024-03-25T19:45:46.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.714+0000 7ff803dfe640 1 librados: init done 2024-03-25T19:45:46.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.715+0000 7ff803dfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:46.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.715+0000 7ff803dfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:46.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.715+0000 7ff803dfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:46.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:46.715+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc0627b0 con 0x7ff7fc13f710 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7ec001220 con 0x7ff7fc13f710 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7ff7ec004c30 con 0x7ff7fc13f710 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f27fc640 10 monclient: dump: 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:47.044 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.044+0000 7ff7f3fff640 4 set_mon_vals no callback set 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.045+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7ff7ec01ac40 con 0x7ff7fc13f710 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.046+0000 7ff801b9a640 10 monclient: get_auth_request con 0x7ff7e006de20 auth_method 0 2024-03-25T19:45:47.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.046+0000 7ff7f27fc640 10 client.4709.objecter ms_handle_connect 0x7ff7e006de20 2024-03-25T19:45:47.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7ff7ec0547e0 con 0x7ff7fc13f710 2024-03-25T19:45:47.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 10 client.4709.objecter ms_dispatch 0x7ff7fc000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:47.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 3 client.4709.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:47.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 3 client.4709.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:47.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-25T19:45:47.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7ec050120 con 0x7ff7fc13f710 2024-03-25T19:45:47.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:47.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.053+0000 7ff7f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:47.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.316+0000 7ff803dfe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:47.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.316+0000 7ff803dfe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:47.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.316+0000 7ff803dfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:45:47.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.316+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7fc14e8b0 con 0x7ff7fc13f710 2024-03-25T19:45:47.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.339+0000 7ff7f27fc640 1 -- 172.21.15.47:0/355980238 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7ff7ec052220 con 0x7ff7fc13f710 2024-03-25T19:45:47.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.339+0000 7ff7f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:47.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.339+0000 7ff7f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:47.339 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:47.339 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 10 librados: watch_flush enter 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 10 librados: watch_flush exit 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 10 client.4709.objecter successfully canceled tick 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 >> v1:172.21.15.142:6800/1025302268 conn(0x7ff7e006de20 legacy=0x7ff7e00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 10 monclient: shutdown 2024-03-25T19:45:47.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:47.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 >> v1:172.21.15.47:6790/0 conn(0x7ff7fc13f710 legacy=0x7ff7fc1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 shutdown_connections 2024-03-25T19:45:47.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 >> 172.21.15.47:0/355980238 conn(0x7ff7fc0b7ea0 msgr2=0x7ff7fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:47.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 shutdown_connections 2024-03-25T19:45:47.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 -- 172.21.15.47:0/355980238 wait complete. 2024-03-25T19:45:47.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:47.341+0000 7ff803dfe640 1 librados: shutdown 2024-03-25T19:45:47.353 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:54.158 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:54.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.275+0000 7fb96a96b640 10 monclient: get_monmap_and_config 2024-03-25T19:45:54.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.275+0000 7fb96a96b640 10 monclient: build_initial_monmap 2024-03-25T19:45:54.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.275+0000 7fb96a96b640 10 monclient: monmap: 2024-03-25T19:45:54.275 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:54.275 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:45:54.276676+0000 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:45:54.276676+0000 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:45:54.276 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.276+0000 7fb96a96b640 1 Processor -- start 2024-03-25T19:45:54.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.276+0000 7fb96a96b640 1 -- start start 2024-03-25T19:45:54.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.276+0000 7fb96a96b640 10 monclient: init 2024-03-25T19:45:54.277 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient(hunting): picked mon.noname-c con 0x7fb9640b4f10 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient(hunting): picked mon.noname-a con 0x7fb9640b44d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient(hunting): picked mon.noname-b con 0x7fb96413f740 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9640b0140 con 0x7fb9640b44d0 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9640af3a0 con 0x7fb9640b4f10 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9640af530 con 0x7fb96413f740 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:54.278 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.277+0000 7fb96a96b640 10 monclient(hunting): authenticate will time out at 6269.563798s 2024-03-25T19:45:54.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.295+0000 7fb9637fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb9640b44d0 0x7fb9640b16c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60712/0 (socket says 172.21.15.47:60712) 2024-03-25T19:45:54.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.295+0000 7fb9637fe640 1 -- 172.21.15.47:0/791345789 learned_addr learned my addr 172.21.15.47:0/791345789 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.317+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128066815 0 0) 0x7fb9640b0140 con 0x7fb9640b44d0 2024-03-25T19:45:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.317+0000 7fb962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.317+0000 7fb962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.317+0000 7fb962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.317+0000 7fb962ffd640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:45:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.317+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb94c003520 con 0x7fb9640b44d0 2024-03-25T19:45:54.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.324+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232629214 0 0) 0x7fb9640af3a0 con 0x7fb9640b4f10 2024-03-25T19:45:54.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.324+0000 7fb962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:54.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.324+0000 7fb962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:54.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.324+0000 7fb962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:54.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.324+0000 7fb962ffd640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:45:54.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.324+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9640b0140 con 0x7fb9640b4f10 2024-03-25T19:45:54.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.325+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139854260 0 0) 0x7fb9640af530 con 0x7fb96413f740 2024-03-25T19:45:54.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.325+0000 7fb962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:54.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.325+0000 7fb962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:54.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.325+0000 7fb962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:54.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.325+0000 7fb962ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:45:54.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.325+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9640af3a0 con 0x7fb96413f740 2024-03-25T19:45:54.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780529734 0 0) 0x7fb94c003520 con 0x7fb9640b44d0 2024-03-25T19:45:54.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 >> v1:172.21.15.47:6790/0 conn(0x7fb96413f740 legacy=0x7fb96413fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:54.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 >> v1:172.21.15.142:6789/0 conn(0x7fb9640b4f10 legacy=0x7fb9640ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:54.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 1 monclient: found mon.noname-a 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9640afcd0 con 0x7fb9640b44d0 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb958001690 con 0x7fb9640b44d0 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb96a96b640 5 monclient: authenticate success, global_id 4728 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb962ffd640 10 monclient: dump: 2024-03-25T19:45:54.348 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.347+0000 7fb96a96b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:54.365 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9580017f0 con 0x7fb9640b44d0 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb962ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb962ffd640 1 -- 172.21.15.47:0/791345789 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb958001c00 con 0x7fb9640b44d0 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb96a96b640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb96a96b640 4 set_mon_vals no callback set 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb962ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.366+0000 7fb962ffd640 10 monclient: dump: 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.366 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:54.367 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:54.367 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:54.367 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:54.367 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:54.367 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 10 monclient: shutdown 2024-03-25T19:45:54.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:54.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 1 -- 172.21.15.47:0/791345789 >> v1:172.21.15.47:6789/0 conn(0x7fb9640b44d0 legacy=0x7fb9640b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:54.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 1 -- 172.21.15.47:0/791345789 shutdown_connections 2024-03-25T19:45:54.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 1 -- 172.21.15.47:0/791345789 >> 172.21.15.47:0/791345789 conn(0x7fb9640617a0 msgr2=0x7fb964061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 1 -- 172.21.15.47:0/791345789 shutdown_connections 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.367+0000 7fb96a96b640 1 -- 172.21.15.47:0/791345789 wait complete. 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.368+0000 7fb96a96b640 10 monclient: build_initial_monmap 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.368+0000 7fb96a96b640 10 monclient: monmap: 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:45:54.369 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.368+0000 7fb96a96b640 1 librados: starting msgr at 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.368+0000 7fb96a96b640 1 librados: starting objecter 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.368+0000 7fb96a96b640 1 Processor -- start 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.369+0000 7fb96a96b640 1 -- start start 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.369+0000 7fb96a96b640 1 librados: setting wanted keys 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.369+0000 7fb96a96b640 1 librados: calling monclient init 2024-03-25T19:45:54.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.369+0000 7fb96a96b640 10 monclient: init 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient(hunting): picked mon.noname-a con 0x7fb9640b44d0 addr v1:172.21.15.47:6789/0 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient(hunting): picked mon.noname-b con 0x7fb9640b4f10 addr v1:172.21.15.142:6789/0 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient(hunting): picked mon.noname-c con 0x7fb96413f740 addr v1:172.21.15.47:6790/0 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9641d51b0 con 0x7fb9640b44d0 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb964131430 con 0x7fb9640b4f10 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb964130040 con 0x7fb96413f740 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:54.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.370+0000 7fb96a96b640 10 monclient(hunting): authenticate will time out at 6269.656309s 2024-03-25T19:45:54.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.376+0000 7fb9637fe640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb9640b4f10 0x7fb9641db440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:57428/0 (socket says 172.21.15.47:57428) 2024-03-25T19:45:54.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.376+0000 7fb9637fe640 1 -- 172.21.15.47:0/2251375127 learned_addr learned my addr 172.21.15.47:0/2251375127 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:45:54.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.382+0000 7fb960ff9640 10 client.?.objecter ms_handle_connect 0x7fb9640b4f10 2024-03-25T19:45:54.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.382+0000 7fb960ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:54.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.382+0000 7fb960ff9640 10 client.?.objecter ms_handle_connect 0x7fb9640b44d0 2024-03-25T19:45:54.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.382+0000 7fb960ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:54.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 10 client.?.objecter ms_handle_connect 0x7fb96413f740 2024-03-25T19:45:54.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:54.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329317965 0 0) 0x7fb964131430 con 0x7fb9640b4f10 2024-03-25T19:45:54.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:54.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:54.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:54.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.395+0000 7fb960ff9640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:45:54.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.396+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb950004eb0 con 0x7fb9640b4f10 2024-03-25T19:45:54.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.397+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995092599 0 0) 0x7fb9641d51b0 con 0x7fb9640b44d0 2024-03-25T19:45:54.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.397+0000 7fb960ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:54.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.397+0000 7fb960ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:54.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.397+0000 7fb960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:54.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.397+0000 7fb960ff9640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:45:54.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.397+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb964131430 con 0x7fb9640b44d0 2024-03-25T19:45:54.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.412+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176789672 0 0) 0x7fb964130040 con 0x7fb96413f740 2024-03-25T19:45:54.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.412+0000 7fb960ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:54.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.412+0000 7fb960ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:54.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.412+0000 7fb960ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:54.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.412+0000 7fb960ff9640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:45:54.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.412+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9641d51b0 con 0x7fb96413f740 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699995118 0 0) 0x7fb964131430 con 0x7fb9640b44d0 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb964130040 con 0x7fb9640b44d0 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb954001970 con 0x7fb9640b44d0 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.423 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793881471 0 0) 0x7fb950004eb0 con 0x7fb9640b4f10 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb964131430 con 0x7fb9640b4f10 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb958001bc0 con 0x7fb9640b4f10 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.423+0000 7fb960ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.424 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.425 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:54.425 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:54.425 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:54.425 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:54.425 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:54.425 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025065035 0 0) 0x7fb964130040 con 0x7fb9640b44d0 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 >> v1:172.21.15.47:6790/0 conn(0x7fb96413f740 legacy=0x7fb9641e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 >> v1:172.21.15.142:6789/0 conn(0x7fb9640b4f10 legacy=0x7fb9641db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 1 monclient: found mon.a 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9641301d0 con 0x7fb9640b44d0 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:54.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb960ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 5 monclient: authenticate success, global_id 4731 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 10 monclient: _renew_subs 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9640ad400 con 0x7fb9640b44d0 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 10 monclient: _renew_subs 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9641e3510 con 0x7fb9640b44d0 2024-03-25T19:45:54.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.443+0000 7fb96a96b640 1 librados: init done 2024-03-25T19:45:54.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.444+0000 7fb96a96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:54.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.444+0000 7fb96a96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:54.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.444+0000 7fb96a96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:54.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.444+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9640ae300 con 0x7fb9640b44d0 2024-03-25T19:45:54.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb954003610 con 0x7fb9640b44d0 2024-03-25T19:45:54.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb960ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:54.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fb954003920 con 0x7fb9640b44d0 2024-03-25T19:45:54.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb960ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:54.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb960ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb960ff9640 10 monclient: dump: 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:45:54.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.785+0000 7fb9627fc640 4 set_mon_vals no callback set 2024-03-25T19:45:54.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.786+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fb9540198f0 con 0x7fb9640b44d0 2024-03-25T19:45:54.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.787+0000 7fb9637fe640 10 monclient: get_auth_request con 0x7fb95006df70 auth_method 0 2024-03-25T19:45:54.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.787+0000 7fb960ff9640 10 client.4731.objecter ms_handle_connect 0x7fb95006df70 2024-03-25T19:45:54.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7fb954053560 con 0x7fb9640b44d0 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 10 client.4731.objecter ms_dispatch 0x7fb964000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 3 client.4731.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 3 client.4731.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb95404f070 con 0x7fb9640b44d0 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:54.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:54.794+0000 7fb960ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:55.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.020+0000 7fb96a96b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.020+0000 7fb96a96b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.020+0000 7fb96a96b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-25T19:45:55.019 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.020+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb964131290 con 0x7fb9640b44d0 2024-03-25T19:45:55.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.049+0000 7fb960ff9640 1 -- 172.21.15.47:0/2251375127 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7fb964131290 con 0x7fb9640b44d0 2024-03-25T19:45:55.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.049+0000 7fb960ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.049+0000 7fb960ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:55.049 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:45:55.049 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:55.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 10 librados: watch_flush enter 2024-03-25T19:45:55.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 10 librados: watch_flush exit 2024-03-25T19:45:55.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T19:45:55.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 10 client.4731.objecter successfully canceled tick 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 >> v1:172.21.15.142:6800/1025302268 conn(0x7fb95006df70 legacy=0x7fb950070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 10 monclient: shutdown 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 >> v1:172.21.15.47:6789/0 conn(0x7fb9640b44d0 legacy=0x7fb9641dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 shutdown_connections 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.051+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 >> 172.21.15.47:0/2251375127 conn(0x7fb9640617a0 msgr2=0x7fb964074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.052+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 shutdown_connections 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.052+0000 7fb96a96b640 1 -- 172.21.15.47:0/2251375127 wait complete. 2024-03-25T19:45:55.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:45:55.052+0000 7fb96a96b640 1 librados: shutdown 2024-03-25T19:45:55.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:01.870 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.980+0000 7f089ed6b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.980+0000 7f089ed6b640 10 monclient: build_initial_monmap 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.980+0000 7f089ed6b640 10 monclient: monmap: 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:46:01.982169+0000 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:46:01.982169+0000 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:46:01.981 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:01.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.981+0000 7f089ed6b640 1 Processor -- start 2024-03-25T19:46:01.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.981+0000 7f089ed6b640 1 -- start start 2024-03-25T19:46:01.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.981+0000 7f089ed6b640 10 monclient: init 2024-03-25T19:46:01.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:01.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient(hunting): picked mon.noname-a con 0x7f08980a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient(hunting): picked mon.noname-c con 0x7f089813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient(hunting): picked mon.noname-b con 0x7f08981430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0898142520 con 0x7f08980a98f0 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08980b28a0 con 0x7f089813f710 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08980b2a30 con 0x7f08981430c0 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:01.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.982+0000 7f089ed6b640 10 monclient(hunting): authenticate will time out at 6277.269418s 2024-03-25T19:46:01.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.988+0000 7f0897fff640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f089813f710 0x7f08981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:47698/0 (socket says 172.21.15.47:47698) 2024-03-25T19:46:01.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.988+0000 7f0897fff640 1 -- 172.21.15.47:0/3841600291 learned_addr learned my addr 172.21.15.47:0/3841600291 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:46:01.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.998+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331112309 0 0) 0x7f08980b28a0 con 0x7f089813f710 2024-03-25T19:46:01.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.998+0000 7f08977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:01.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.998+0000 7f08977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:01.998 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.998+0000 7f08977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:01.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.998+0000 7f08977fe640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:46:01.999 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:01.998+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0888003520 con 0x7f089813f710 2024-03-25T19:46:02.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037514084 0 0) 0x7f0888003520 con 0x7f089813f710 2024-03-25T19:46:02.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 >> v1:172.21.15.47:6790/0 conn(0x7f08981430c0 legacy=0x7f089814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 >> v1:172.21.15.47:6789/0 conn(0x7f08980a98f0 legacy=0x7f08980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 1 monclient: found mon.noname-c 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.142:6789/0 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980b44c0 con 0x7f089813f710 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f0884001690 con 0x7f089813f710 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:02.009 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.007+0000 7f08977fe640 10 monclient: dump: 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.008+0000 7f089ed6b640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:46:02.010 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.008+0000 7f089ed6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:02.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 <== mon.1 v1:172.21.15.142:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08840017f0 con 0x7f089813f710 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f08977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f08977fe640 1 -- 172.21.15.47:0/3841600291 <== mon.1 v1:172.21.15.142:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f0884001c40 con 0x7f089813f710 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f089ed6b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f089ed6b640 4 set_mon_vals no callback set 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f08977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f08977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.044+0000 7f08977fe640 10 monclient: dump: 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.045 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.046 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:02.046 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:02.046 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:02.046 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:02.046 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:02.046 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:02.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 10 monclient: shutdown 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 1 -- 172.21.15.47:0/3841600291 >> v1:172.21.15.142:6789/0 conn(0x7f089813f710 legacy=0x7f08981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 1 -- 172.21.15.47:0/3841600291 shutdown_connections 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 1 -- 172.21.15.47:0/3841600291 >> 172.21.15.47:0/3841600291 conn(0x7f08980b7ea0 msgr2=0x7f08980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 1 -- 172.21.15.47:0/3841600291 shutdown_connections 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.045+0000 7f089ed6b640 1 -- 172.21.15.47:0/3841600291 wait complete. 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 10 monclient: build_initial_monmap 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 10 monclient: monmap: 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:46:02.049 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 1 librados: starting msgr at 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 1 librados: starting objecter 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 1 Processor -- start 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 1 -- start start 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 1 librados: setting wanted keys 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 1 librados: calling monclient init 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.046+0000 7f089ed6b640 10 monclient: init 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:02.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient(hunting): picked mon.noname-c con 0x7f08980a98f0 addr v1:172.21.15.47:6790/0 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient(hunting): picked mon.noname-a con 0x7f089813f710 addr v1:172.21.15.47:6789/0 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient(hunting): picked mon.noname-b con 0x7f08981430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0898137df0 con 0x7f089813f710 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08981337a0 con 0x7f08981430c0 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0898133930 con 0x7f08980a98f0 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:02.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.047+0000 7f089ed6b640 10 monclient(hunting): authenticate will time out at 6277.334563s 2024-03-25T19:46:02.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.061+0000 7f089d308640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f08981430c0 0x7f0898133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:47736/0 (socket says 172.21.15.47:47736) 2024-03-25T19:46:02.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.061+0000 7f089d308640 1 -- 172.21.15.47:0/1177861944 learned_addr learned my addr 172.21.15.47:0/1177861944 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:46:02.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.062+0000 7f08957fa640 10 client.?.objecter ms_handle_connect 0x7f08981430c0 2024-03-25T19:46:02.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.062+0000 7f08957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:02.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.067+0000 7f08957fa640 10 client.?.objecter ms_handle_connect 0x7f08980a98f0 2024-03-25T19:46:02.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.067+0000 7f08957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:02.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.070+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210426114 0 0) 0x7f0898133930 con 0x7f08980a98f0 2024-03-25T19:46:02.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.070+0000 7f08957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:02.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.070+0000 7f08957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:02.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.070+0000 7f08957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:02.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.070+0000 7f08957fa640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:46:02.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.070+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c004eb0 con 0x7f08980a98f0 2024-03-25T19:46:02.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.084+0000 7f08957fa640 10 client.?.objecter ms_handle_connect 0x7f089813f710 2024-03-25T19:46:02.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.084+0000 7f08957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762341384 0 0) 0x7f087c004eb0 con 0x7f08980a98f0 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0898133930 con 0x7f08980a98f0 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f088c001930 con 0x7f08980a98f0 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 10 monclient(hunting): dump: 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.087 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.086+0000 7f08957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.087+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544740157 0 0) 0x7f08981337a0 con 0x7f08981430c0 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.087+0000 7f08957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.087+0000 7f08957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:02.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.087+0000 7f08957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:02.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.087+0000 7f08957fa640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:46:02.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.087+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c004eb0 con 0x7f08981430c0 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1527215138 0 0) 0x7f0898133930 con 0x7f08980a98f0 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 >> v1:172.21.15.142:6789/0 conn(0x7f08981430c0 legacy=0x7f0898133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 >> v1:172.21.15.47:6789/0 conn(0x7f089813f710 legacy=0x7f08981376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 1 monclient: found mon.c 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08981367d0 con 0x7f08980a98f0 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:02.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f08957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 5 monclient: authenticate success, global_id 4721 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 10 monclient: _renew_subs 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08981369e0 con 0x7f08980a98f0 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 10 client.4721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 10 monclient: _renew_subs 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f089813b640 con 0x7f08980a98f0 2024-03-25T19:46:02.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.096+0000 7f089ed6b640 1 librados: init done 2024-03-25T19:46:02.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.097+0000 7f089ed6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:02.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.097+0000 7f089ed6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:02.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.097+0000 7f089ed6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:46:02.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.097+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08980627b0 con 0x7f08980a98f0 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f088c003610 con 0x7f08980a98f0 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f08957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f0896ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7f088c0039e0 con 0x7f08980a98f0 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f08957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f08957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.121+0000 7f08957fa640 10 monclient: dump: 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.123 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.122+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7f088c003b40 con 0x7f08980a98f0 2024-03-25T19:46:02.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.123+0000 7f0897fff640 10 monclient: get_auth_request con 0x7f087c06df80 auth_method 0 2024-03-25T19:46:02.125 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.123+0000 7f08957fa640 10 client.4721.objecter ms_handle_connect 0x7f087c06df80 2024-03-25T19:46:02.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.424+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7f088c053460 con 0x7f08980a98f0 2024-03-25T19:46:02.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.424+0000 7f08957fa640 10 client.4721.objecter ms_dispatch 0x7f0898000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:46:02.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.424+0000 7f08957fa640 3 client.4721.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:46:02.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.424+0000 7f08957fa640 3 client.4721.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:46:02.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.424+0000 7f08957fa640 20 client.4721.objecter dump_active .. 0 homeless 2024-03-25T19:46:02.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.424+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f088c04eda0 con 0x7f08980a98f0 2024-03-25T19:46:02.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.425+0000 7f08957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:02.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.425+0000 7f08957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:02.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.654+0000 7f089ed6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:02.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.654+0000 7f089ed6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:02.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.654+0000 7f089ed6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-25T19:46:02.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.654+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0898137300 con 0x7f08980a98f0 2024-03-25T19:46:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.805+0000 7f08957fa640 1 -- 172.21.15.47:0/1177861944 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7f088c050ea0 con 0x7f08980a98f0 2024-03-25T19:46:02.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.805+0000 7f08957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:02.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.805+0000 7f08957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:02.806 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:46:02.806 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:02.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.807+0000 7f089ed6b640 10 librados: watch_flush enter 2024-03-25T19:46:02.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.807+0000 7f089ed6b640 10 librados: watch_flush exit 2024-03-25T19:46:02.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.807+0000 7f089ed6b640 20 client.4721.objecter shutdown clearing up homeless session... 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.807+0000 7f089ed6b640 10 client.4721.objecter successfully canceled tick 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 >> v1:172.21.15.142:6800/1025302268 conn(0x7f087c06df80 legacy=0x7f087c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 10 monclient: shutdown 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 >> v1:172.21.15.47:6790/0 conn(0x7f08980a98f0 legacy=0x7f08981392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 shutdown_connections 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 >> 172.21.15.47:0/1177861944 conn(0x7f08980b7ea0 msgr2=0x7f0898074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 shutdown_connections 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 -- 172.21.15.47:0/1177861944 wait complete. 2024-03-25T19:46:02.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:02.808+0000 7f089ed6b640 1 librados: shutdown 2024-03-25T19:46:02.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:04.193 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:41.428 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.249 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:41.484 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.279 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:41.515 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.311 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:41.547 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.312 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:41.501 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.318 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:41.508 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.319 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:41.555 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.340 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:11:41.529 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:04.351 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:41.587 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:09.628 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:09.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.732+0000 7fc37f450640 10 monclient: get_monmap_and_config 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.732+0000 7fc37f450640 10 monclient: build_initial_monmap 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.732+0000 7fc37f450640 10 monclient: monmap: 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:46:09.733860+0000 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:46:09.733860+0000 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-c 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-25T19:46:09.735 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.732+0000 7fc37f450640 1 Processor -- start 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 1 -- start start 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 10 monclient: init 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 10 monclient(hunting): picked mon.noname-a con 0x7fc3780a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 10 monclient(hunting): picked mon.noname-b con 0x7fc37813f710 addr v1:172.21.15.47:6790/0 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 10 monclient(hunting): picked mon.noname-c con 0x7fc3781430c0 addr v1:172.21.15.142:6789/0 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc378142520 con 0x7fc3780a98f0 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.733+0000 7fc37f450640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780b28a0 con 0x7fc3781430c0 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.734+0000 7fc37f450640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780b2a30 con 0x7fc37813f710 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.734+0000 7fc37f450640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:09.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.734+0000 7fc37f450640 10 monclient(hunting): authenticate will time out at 6285.020979s 2024-03-25T19:46:09.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.746+0000 7fc37d9ed640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc3781430c0 0x7fc37814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:55196/0 (socket says 172.21.15.47:55196) 2024-03-25T19:46:09.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.746+0000 7fc37d9ed640 1 -- 172.21.15.47:0/1025905885 learned_addr learned my addr 172.21.15.47:0/1025905885 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:46:09.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.758+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905735143 0 0) 0x7fc3780b2a30 con 0x7fc37813f710 2024-03-25T19:46:09.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.758+0000 7fc36ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:09.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.758+0000 7fc36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:09.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.758+0000 7fc36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:09.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.758+0000 7fc36ffff640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:46:09.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.758+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc364003520 con 0x7fc37813f710 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997606352 0 0) 0x7fc378142520 con 0x7fc3780a98f0 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3780b2a30 con 0x7fc3780a98f0 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002573709 0 0) 0x7fc3780b28a0 con 0x7fc3781430c0 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:09.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:46:09.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.769+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc378142520 con 0x7fc3781430c0 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687336413 0 0) 0x7fc364003520 con 0x7fc37813f710 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 >> v1:172.21.15.142:6789/0 conn(0x7fc3781430c0 legacy=0x7fc37814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 >> v1:172.21.15.47:6789/0 conn(0x7fc3780a98f0 legacy=0x7fc3780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 1 monclient: found mon.noname-b 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780b44c0 con 0x7fc37813f710 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: _finish_auth 0 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:09.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc368001690 con 0x7fc37813f710 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc36ffff640 10 monclient: dump: 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:09.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc37f450640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:46:09.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.788+0000 7fc37f450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3680017f0 con 0x7fc37813f710 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc36ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc36ffff640 1 -- 172.21.15.47:0/1025905885 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc368001c00 con 0x7fc37813f710 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc37f450640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc37f450640 4 set_mon_vals no callback set 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc36ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc36ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.818+0000 7fc36ffff640 10 monclient: dump: 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.819 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:09.820 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:09.820 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:09.820 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:09.820 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:09.820 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 10 monclient: shutdown 2024-03-25T19:46:09.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 1 -- 172.21.15.47:0/1025905885 >> v1:172.21.15.47:6790/0 conn(0x7fc37813f710 legacy=0x7fc3781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 1 -- 172.21.15.47:0/1025905885 shutdown_connections 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 1 -- 172.21.15.47:0/1025905885 >> 172.21.15.47:0/1025905885 conn(0x7fc3780b7ea0 msgr2=0x7fc3780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 1 -- 172.21.15.47:0/1025905885 shutdown_connections 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.819+0000 7fc37f450640 1 -- 172.21.15.47:0/1025905885 wait complete. 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.820+0000 7fc37f450640 10 monclient: build_initial_monmap 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.820+0000 7fc37f450640 10 monclient: monmap: 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:09.822 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.noname-b 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 1 librados: starting msgr at 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 1 librados: starting objecter 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 1 Processor -- start 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 1 -- start start 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 1 librados: setting wanted keys 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 1 librados: calling monclient init 2024-03-25T19:46:09.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 10 monclient: init 2024-03-25T19:46:09.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.821+0000 7fc37f450640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:09.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:09.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 10 monclient(hunting): picked mon.noname-a con 0x7fc3780a98f0 addr v1:172.21.15.47:6789/0 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 10 monclient(hunting): picked mon.noname-b con 0x7fc37813f710 addr v1:172.21.15.142:6789/0 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 10 monclient(hunting): picked mon.noname-c con 0x7fc3781430c0 addr v1:172.21.15.47:6790/0 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc378137df0 con 0x7fc3780a98f0 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3781337a0 con 0x7fc37813f710 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc378133930 con 0x7fc3781430c0 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:09.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.822+0000 7fc37f450640 10 monclient(hunting): authenticate will time out at 6285.109270s 2024-03-25T19:46:09.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.837+0000 7fc37c9eb640 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc37813f710 0x7fc3781376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.47:55256/0 (socket says 172.21.15.47:55256) 2024-03-25T19:46:09.837 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.837+0000 7fc37c9eb640 1 -- 172.21.15.47:0/3431753745 learned_addr learned my addr 172.21.15.47:0/3431753745 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-25T19:46:09.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.840+0000 7fc36dffb640 10 client.?.objecter ms_handle_connect 0x7fc37813f710 2024-03-25T19:46:09.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.840+0000 7fc36dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:09.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.843+0000 7fc36dffb640 10 client.?.objecter ms_handle_connect 0x7fc3780a98f0 2024-03-25T19:46:09.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.843+0000 7fc36dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:09.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.859+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161488404 0 0) 0x7fc378137df0 con 0x7fc3780a98f0 2024-03-25T19:46:09.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.859+0000 7fc36dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:09.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.859+0000 7fc36dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:09.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.859+0000 7fc36dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:09.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.859+0000 7fc36dffb640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:46:09.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.859+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c004eb0 con 0x7fc3780a98f0 2024-03-25T19:46:09.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.860+0000 7fc36dffb640 10 client.?.objecter ms_handle_connect 0x7fc3781430c0 2024-03-25T19:46:09.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.860+0000 7fc36dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:09.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.866+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136720114 0 0) 0x7fc3781337a0 con 0x7fc37813f710 2024-03-25T19:46:09.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.866+0000 7fc36dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:09.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.866+0000 7fc36dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:09.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.866+0000 7fc36dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:09.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.866+0000 7fc36dffb640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:46:09.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.866+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc378137df0 con 0x7fc37813f710 2024-03-25T19:46:09.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103862410 0 0) 0x7fc378137df0 con 0x7fc37813f710 2024-03-25T19:46:09.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3781337a0 con 0x7fc37813f710 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc368001b40 con 0x7fc37813f710 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 10 monclient(hunting): dump: 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:09.891 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.890+0000 7fc36dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.891+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1674949465 0 0) 0x7fc35c004eb0 con 0x7fc3780a98f0 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.891+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc378137df0 con 0x7fc3780a98f0 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.891+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc374001970 con 0x7fc3780a98f0 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.891+0000 7fc36dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.891+0000 7fc36dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.891+0000 7fc36dffb640 10 monclient(hunting): dump: 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.892 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:09.893 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:09.893 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:09.893 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:09.893 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:09.893 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:09.893 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2234570317 0 0) 0x7fc3781337a0 con 0x7fc37813f710 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 >> v1:172.21.15.47:6790/0 conn(0x7fc3781430c0 legacy=0x7fc378133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 >> v1:172.21.15.47:6789/0 conn(0x7fc3780a98f0 legacy=0x7fc3781392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 1 monclient: found mon.b 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:46:09.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3781367d0 con 0x7fc37813f710 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc36dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 10 monclient: _renew_subs 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3781369e0 con 0x7fc37813f710 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 10 monclient: _renew_subs 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37813b640 con 0x7fc37813f710 2024-03-25T19:46:09.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.896+0000 7fc37f450640 1 librados: init done 2024-03-25T19:46:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.897+0000 7fc37f450640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.897+0000 7fc37f450640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.897+0000 7fc37f450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:46:09.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:09.897+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3780627b0 con 0x7fc37813f710 2024-03-25T19:46:10.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3680011f0 con 0x7fc37813f710 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3263226010 0 0) 0x7fc368003a40 con 0x7fc37813f710 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36dffb640 10 monclient: dump: 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:fsid ff41e6e4-140a-4dba-9564-ead70286575a 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-25T19:33:51.808098+0000 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 3 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.142:6789/0 mon.b 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-25T19:46:10.211 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-25T19:46:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.210+0000 7fc36f7fe640 4 set_mon_vals no callback set 2024-03-25T19:46:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.211+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1027017634 0 0) 0x7fc368003c60 con 0x7fc37813f710 2024-03-25T19:46:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.211+0000 7fc37d1ec640 10 monclient: get_auth_request con 0x7fc35c06de40 auth_method 0 2024-03-25T19:46:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.212+0000 7fc36dffb640 10 client.4735.objecter ms_handle_connect 0x7fc35c06de40 2024-03-25T19:46:10.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.217+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 546990502 0 0) 0x7fc3680547f0 con 0x7fc37813f710 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.217+0000 7fc36dffb640 10 client.4735.objecter ms_dispatch 0x7fc378000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.217+0000 7fc36dffb640 3 client.4735.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.217+0000 7fc36dffb640 3 client.4735.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.217+0000 7fc36dffb640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.218+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc368050130 con 0x7fc37813f710 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.218+0000 7fc36dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:10.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.218+0000 7fc36dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:10.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.480+0000 7fc37f450640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:10.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.480+0000 7fc37f450640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:10.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.480+0000 7fc37f450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.142:6789/0 2024-03-25T19:46:10.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.480+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 --> v1:172.21.15.142:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc378074cc0 con 0x7fc37813f710 2024-03-25T19:46:10.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.511+0000 7fc36dffb640 1 -- 172.21.15.47:0/3431753745 <== mon.1 v1:172.21.15.142:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 1582841953) 0x7fc368052230 con 0x7fc37813f710 2024-03-25T19:46:10.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.511+0000 7fc36dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:10.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.511+0000 7fc36dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:10.512 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-25T19:46:10.512 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"ff41e6e4-140a-4dba-9564-ead70286575a","created":"2024-03-25T19:34:40.343304+0000","modified":"2024-03-25T19:44:48.930621+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:44:48.930621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24d0b8fc-9364-43bb-9d61-e0b8db08e324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"27159e00-50f8-4194-9d01-5726783de16d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3f06d608-135b-4afa-a600-7d20cd766e57","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6457e632-00d5-4e26-9170-109d2d2fd4e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90d5c46a-9060-4cbc-82d5-7bed26691884","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c80800cd-ba3d-4a90-96fa-f3ba622e9836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ee8e69c9-9b53-4daa-a98d-8b481464375a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"451b8a1c-8ed2-41d1-9fff-16fe77956190","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 10 librados: watch_flush enter 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 10 librados: watch_flush exit 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 10 client.4735.objecter successfully canceled tick 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 >> v1:172.21.15.142:6800/1025302268 conn(0x7fc35c06de40 legacy=0x7fc35c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 10 monclient: shutdown 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 >> v1:172.21.15.142:6789/0 conn(0x7fc37813f710 legacy=0x7fc3781376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 shutdown_connections 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 >> 172.21.15.47:0/3431753745 conn(0x7fc3780b7ea0 msgr2=0x7fc378074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 shutdown_connections 2024-03-25T19:46:10.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 -- 172.21.15.47:0/3431753745 wait complete. 2024-03-25T19:46:10.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-25T19:46:10.514+0000 7fc37f450640 1 librados: shutdown 2024-03-25T19:46:10.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:10.526 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:46:10.528 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:46:10.528 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:46:10.528 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:12.440 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:11:25.931 35558== 2024-03-25T19:46:12.440 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:11:25.931 35558== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:12.475 INFO:tasks.ceph.osd.0.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:16.630 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:46:16.631 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:46:16.631 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:46:16.632 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:18.647 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:11:32.124 35559== 2024-03-25T19:46:18.647 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:11:32.124 35559== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:18.688 INFO:tasks.ceph.osd.1.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:22.734 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:46:22.734 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:46:22.735 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:46:22.735 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:24.301 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:01.537 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.310 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:01.546 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.340 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:01.576 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.352 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:12:01.542 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.361 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:12:01.551 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.366 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:01.601 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.370 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:01.606 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.388 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:12:01.578 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.402 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:01.637 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:24.519 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:11:37.996 35560== 2024-03-25T19:46:24.519 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:11:37.996 35560== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:24.566 INFO:tasks.ceph.osd.2.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:28.836 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:46:28.837 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:46:28.837 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:46:28.837 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:30.539 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:11:44.011 35561== 2024-03-25T19:46:30.539 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:11:44.011 35561== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:30.574 INFO:tasks.ceph.osd.3.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:34.939 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:46:34.939 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:46:34.940 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:46:34.940 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:36.868 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:11:50.362 35553== 2024-03-25T19:46:36.869 INFO:tasks.ceph.osd.4.smithi142.stderr:==00:00:11:50.362 35553== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:36.904 INFO:tasks.ceph.osd.4.smithi142.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:41.041 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:46:41.042 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:46:41.042 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:46:41.042 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:43.044 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:11:56.538 35554== 2024-03-25T19:46:43.044 INFO:tasks.ceph.osd.5.smithi142.stderr:==00:00:11:56.538 35554== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:43.083 INFO:tasks.ceph.osd.5.smithi142.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:44.466 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:21.702 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.470 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:21.705 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.497 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:21.732 35244== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.515 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:12:21.705 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.522 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:12:21.712 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.529 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:21.765 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.535 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:12:21.725 34970== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.538 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:21.774 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:44.556 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:21.791 35248== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:47.144 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:46:47.144 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:46:47.145 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:46:47.145 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:48.837 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:12:02.330 35555== 2024-03-25T19:46:48.837 INFO:tasks.ceph.osd.6.smithi142.stderr:==00:00:12:02.330 35555== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:48.884 INFO:tasks.ceph.osd.6.smithi142.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:53.246 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:46:53.247 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:46:53.247 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:46:53.247 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:54.692 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:12:08.180 35556== 2024-03-25T19:46:54.693 INFO:tasks.ceph.osd.7.smithi142.stderr:==00:00:12:08.180 35556== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:54.722 INFO:tasks.ceph.osd.7.smithi142.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:46:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:46:59.349 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:46:59.350 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:46:59.350 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:46:59.350 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:46:59.351 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:59.384 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:46:59.384 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:46:59.384 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:59.385 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:46:59.385+0000 7fae4ad01640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34968) UID: 0 2024-03-25T19:46:59.386 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-03-25T19:46:59.385+0000 7fae4ad01640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:46:59.424 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:46:59.425 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:46:59.425 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:46:59.425 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:59.447 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-03-25T19:46:59.435+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35213) UID: 0 2024-03-25T19:46:59.452 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-03-25T19:46:59.436+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:46:59.874 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:37.109 35244== 2024-03-25T19:46:59.874 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:37.109 35244== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:59.884 INFO:tasks.ceph.mon.a.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:05.527 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:47:05.527 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:47:05.527 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:47:05.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:05.543 INFO:tasks.ceph.mon.c.smithi047.stderr:2024-03-25T19:47:05.532+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35215) UID: 0 2024-03-25T19:47:05.544 INFO:tasks.ceph.mon.c.smithi047.stderr:2024-03-25T19:47:05.534+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:49:01.056 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:38.291 35248== 2024-03-25T19:49:01.057 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:38.291 35248== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:49:01.065 INFO:tasks.ceph.mon.c.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:49:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:49:01.456 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:49:01.457 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:49:01.457 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:49:01.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:49:01.473 INFO:tasks.ceph.mon.b.smithi142.stderr:2024-03-25T19:49:01.462+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34966) UID: 0 2024-03-25T19:49:01.473 INFO:tasks.ceph.mon.b.smithi142.stderr:2024-03-25T19:49:01.463+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:49:01.970 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:14:39.160 34970== 2024-03-25T19:49:01.970 INFO:tasks.ceph.mon.b.smithi142.stderr:==00:00:14:39.160 34970== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:49:01.979 INFO:tasks.ceph.mon.b.smithi142.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:49:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:49:07.559 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:49:07.559 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:49:07.560 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:49:07.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:49:07.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:49:07.609 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:49:07.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:49:07.703 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:49:07.787 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:49:07.787 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:49:07.844 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:49:07.844 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:49:07.912 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:49:07.912 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:49:08.037 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:49:08.037 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:49:08.109 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:49:08.110 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:49:08.169 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:49:08.169 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:49:08.225 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:49:08.227 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/data/mon.a.tgz 2024-03-25T19:49:08.227 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-25T19:49:08.242 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.8QGfcQTiLV 2024-03-25T19:49:08.242 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8QGfcQTiLV 2024-03-25T19:49:08.349 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.8QGfcQTiLV 2024-03-25T19:49:08.453 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.8QGfcQTiLV is 134KB 2024-03-25T19:49:08.505 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.8QGfcQTiLV 2024-03-25T19:49:08.519 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/data/mon.c.tgz 2024-03-25T19:49:08.519 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-25T19:49:08.572 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.1OPadNSZ6x 2024-03-25T19:49:08.573 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.1OPadNSZ6x 2024-03-25T19:49:08.661 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.1OPadNSZ6x 2024-03-25T19:49:08.745 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.1OPadNSZ6x is 134KB 2024-03-25T19:49:08.795 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.1OPadNSZ6x 2024-03-25T19:49:08.809 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/data/mon.b.tgz 2024-03-25T19:49:08.809 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-03-25T19:49:08.830 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.lCjAftlTXV 2024-03-25T19:49:08.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.lCjAftlTXV 2024-03-25T19:49:08.933 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.lCjAftlTXV 2024-03-25T19:49:09.020 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.lCjAftlTXV is 135KB 2024-03-25T19:49:09.071 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.lCjAftlTXV 2024-03-25T19:49:09.086 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:49:09.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:49:09.088 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:49:09.476 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:49:09.721 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:49:09.723 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/remote/smithi047/crash 2024-03-25T19:49:09.724 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:49:09.752 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/remote/smithi142/crash 2024-03-25T19:49:09.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:49:09.780 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:49:09.781 DEBUG:teuthology.orchestra.run.smithi047:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:49:09.797 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:49:09.821 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-03-25T19:49:09.821 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39634.log 2024-03-25T19:49:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35283.log 2024-03-25T19:49:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:49:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35283.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:49:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:49:09.869 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35283.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35203.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35243.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35203.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35123.log 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35203.log.gz 2024-03-25T19:49:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35163.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35243.log.gz 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35123.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35323.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35123.log.gz 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-osd.4.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35163.log.gz 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35414.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35323.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35414.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35323.log.gz 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-osd.7.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35414.log.gz 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35374.log 2024-03-25T19:49:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:49:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35374.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:49:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-mgr.x.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35374.log.gz 2024-03-25T19:49:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:49:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:49:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41082.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:49:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40652.log 2024-03-25T19:49:09.878 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-03-25T19:49:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39673.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39634.log.gz 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:gzip/var/log/ceph/ceph-client.admin.40652.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41979.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40652.log.gz 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41979.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39673.log.gz 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42057.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39790.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40375.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41979.log.gz 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40453.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39790.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.42057.log.gz 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:gzip/var/log/ceph/ceph-client.admin.40453.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39122.log 2024-03-25T19:49:09.879 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39790.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40219.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39122.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40453.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39122.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:49:09.880 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39242.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39242.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39242.log.gz 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40613.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29389.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38874.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42096.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.tmp-client.admin.29389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35255.log 2024-03-25T19:49:09.881 INFO:teuthology.orchestra.run.smithi047.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29389.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.35255.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42096.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41862.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41238.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35255.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40769.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41862.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42369.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41862.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40847.log 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40769.log.gz 2024-03-25T19:49:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39985.log 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40847.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42369.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40847.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41901.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39985.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41589.log 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39829.log 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35557.log 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39829.log: gzip 86.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.41589.log.gz 2024-03-25T19:49:09.883 INFO:teuthology.orchestra.run.smithi047.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.35557.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39829.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42018.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35557.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41277.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40535.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40886.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42291.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40535.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-03-25T19:49:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42291.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40535.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42135.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42291.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39907.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42135.log: 86.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37630.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39907.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42135.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.37630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39907.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41667.log 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37630.log.gz 2024-03-25T19:49:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42330.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39595.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41667.log.gz 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42330.log.gz 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40102.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41472.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-03-25T19:49:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39868.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39946.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35732.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39868.log.gz 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39946.log.gz 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.35732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41784.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2024-03-25T19:49:09.887 INFO:teuthology.orchestra.run.smithi047.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35732.log.gz 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0%/var/log/ceph/ceph-client.admin.41784.log: -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40808.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40297.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41784.log.gz 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39751.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39712.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40808.log.gz 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39751.log.gz 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41940.log 2024-03-25T19:49:09.888 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39712.log.gz 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41160.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42213.log 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40180.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41940.log.gz 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41511.log 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42213.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41511.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42213.log.gz/var/log/ceph/ceph-client.admin.39398.log: 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-03-25T19:49:09.889 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41511.log.gz 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39515.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39320.log: /var/log/ceph/ceph-client.admin.39515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41004.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1%/var/log/ceph/ceph-client.admin.35584.log: -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40141.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39515.log.gz 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41004.log: /var/log/ceph/ceph-client.admin.40141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36598.log 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41004.log.gz 2024-03-25T19:49:09.890 INFO:teuthology.orchestra.run.smithi047.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40141.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40965.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.36598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41745.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36598.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr: -- /var/log/ceph/ceph-client.admin.40258.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8%/var/log/ceph/ceph-client.admin.40258.log: -- replaced with /var/log/ceph/ceph-client.admin.41745.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42174.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42174.log: /var/log/ceph/ceph-client.admin.39083.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42174.log.gz 2024-03-25T19:49:09.891 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.42252.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz/var/log/ceph/ceph.log: 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40063.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39556.log: /var/log/ceph/ceph-client.admin.40063.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: 92.9% 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40063.log.gz 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39437.log/var/log/ceph/ceph-osd.2.log: 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40730.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3%/var/log/ceph/ceph-client.admin.39437.log: -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40336.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40730.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39437.log.gz 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-03-25T19:49:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40730.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40024.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40336.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39163.log 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:gzip/var/log/ceph/ceph-client.admin.41394.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3%/var/log/ceph/ceph-client.admin.41433.log: -- replaced with /var/log/ceph/ceph-client.admin.39163.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39281.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2024-03-25T19:49:09.893 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2024-03-25T19:49:09.938 INFO:teuthology.orchestra.run.smithi142.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:49:10.673 INFO:teuthology.orchestra.run.smithi047.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:49:10.730 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:49:10.732 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:49:10.748 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:49:10.986 INFO:teuthology.orchestra.run.smithi047.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:49:10.991 INFO:teuthology.orchestra.run.smithi047.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:49:11.078 INFO:teuthology.orchestra.run.smithi047.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:49:11.091 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:49:14.640 INFO:teuthology.orchestra.run.smithi047.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:49:15.127 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:49:15.129 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-03-25T19:49:15.129 INFO:teuthology.orchestra.run.smithi142.stderr:real 0m5.288s 2024-03-25T19:49:15.129 INFO:teuthology.orchestra.run.smithi142.stderr:user 0m8.925s 2024-03-25T19:49:15.129 INFO:teuthology.orchestra.run.smithi142.stderr:sys 0m0.322s 2024-03-25T19:49:15.253 INFO:teuthology.orchestra.run.smithi047.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:49:15.255 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-25T19:49:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:real 0m5.446s 2024-03-25T19:49:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:user 0m14.194s 2024-03-25T19:49:15.256 INFO:teuthology.orchestra.run.smithi047.stderr:sys 0m0.489s 2024-03-25T19:49:15.256 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:49:15.256 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/remote/smithi047/log 2024-03-25T19:49:15.257 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:49:17.479 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/remote/smithi142/log 2024-03-25T19:49:17.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:49:18.918 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:49:19.125 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=96155c27080f47718a01cfe36f22c243 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:49:19.130 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:49:19.139 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:49:19.151 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:49:19.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:49:19.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:49:19.239 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:49:19.248 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:49:19.249 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:49:19.251 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:49:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:49:19.268 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:49:19.270 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:49:19.270 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-03-25T19:49:19.270 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 -480us[ -480us] +/- 75ms 2024-03-25T19:49:19.270 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -265us[ -261us] +/- 62ms 2024-03-25T19:49:19.270 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +720us[ +724us] +/- 65ms 2024-03-25T19:49:19.271 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:49:19.273 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:49:19.274 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-03-25T19:49:19.274 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 -471us[ -471us] +/- 75ms 2024-03-25T19:49:19.274 INFO:teuthology.orchestra.run.smithi142.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -252us[ -256us] +/- 62ms 2024-03-25T19:49:19.274 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 65 +717us[ +717us] +/- 64ms 2024-03-25T19:49:19.274 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:49:19.274 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:49:19.282 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:49:19.283 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:49:19.292 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:49:19.385 DEBUG:teuthology.orchestra.run.smithi142:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:49:19.457 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:49:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:49:19.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:49:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:49:19.521 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:49:19.529 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:49:19.543 INFO:teuthology.task.internal:Duration was 6118.299417 seconds 2024-03-25T19:49:19.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:49:19.558 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:49:19.558 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:49:19.561 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:49:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:49:19.608 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:49:19.985 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:49:19.985 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-03-25T19:49:19.985 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:49:20.006 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-03-25T19:49:20.006 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:49:20.044 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:49:20.044 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:49:20.049 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:49:20.115 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:49:20.116 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:49:20.120 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:49:20.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:49:20.271 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:49:20.272 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:49:20.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:49:20.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:49:20.337 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:49:20.341 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:49:20.364 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-03-25T19:49:20.391 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-03-25T19:49:20.413 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:49:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:49:20.439 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:49:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:49:20.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:49:20.476 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:49:20.477 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/remote/smithi047 2024-03-25T19:49:20.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:49:20.515 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320/remote/smithi142 2024-03-25T19:49:20.516 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:49:20.551 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:49:20.551 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:49:20.553 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:49:20.600 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:49:20.613 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:49:20.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:49:20.628 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:49:20.628 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:49:20.630 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:49:20.642 INFO:teuthology.orchestra.run.smithi047.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:49 /home/ubuntu/cephtest 2024-03-25T19:49:20.655 INFO:teuthology.orchestra.run.smithi142.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:49 /home/ubuntu/cephtest 2024-03-25T19:49:20.656 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:49:20.665 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:49:20.695 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:49:20.717 DEBUG:teuthology.nuke:shortname: smithi047 2024-03-25T19:49:20.717 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:49:20.739 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:58:27.344015', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFN2YZrooIvVsISkt9MTFOIWr/dG0HSHOvYeYXmO+8d07avH0ttJ1lX1LOopxikblA5JwqPr0bIC6G6CEjmTZ5I='} 2024-03-25T19:49:20.743 DEBUG:teuthology.nuke:shortname: smithi142 2024-03-25T19:49:20.743 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:49:20.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:58:27.345894', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFignQ13rXWoi83NQO6kOItIIJTX6k/evnxiJbbCD4HIK+LWHM0J7ITDwhp08F5/ptql0cVFiEQLX4gfn4ga9j4='} 2024-03-25T19:49:20.796 INFO:teuthology.orchestra.console.smithi047:Power off 2024-03-25T19:49:20.796 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:49:20.819 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2024-03-25T19:49:20.819 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:20.834 INFO:teuthology.orchestra.console.smithi142:Power off 2024-03-25T19:49:20.834 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:49:20.844 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-03-25T19:49:20.858 DEBUG:teuthology.orchestra.console.smithi142:power off output: Chassis Power Control: Down/Off 2024-03-25T19:49:20.858 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:20.880 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-03-25T19:49:24.846 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:24.881 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:24.956 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-03-25T19:49:24.990 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-03-25T19:49:28.961 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:28.991 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:29.073 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2024-03-25T19:49:29.102 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-03-25T19:49:29.175 INFO:teuthology.orchestra.console.smithi047:Power off completed 2024-03-25T19:49:33.107 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:49:33.216 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off 2024-03-25T19:49:33.317 INFO:teuthology.orchestra.console.smithi142:Power off completed 2024-03-25T19:49:33.421 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6118.299417257309 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=96155c27080f47718a01cfe36f22c243 status: fail success: false 2024-03-25T19:49:33.422 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:49:33.507 INFO:teuthology.run:FAIL